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Foreword 


Stuart Jeffrey 


Glasgow School of Art 
S Jeffrey@gsa.ac.uk 


This diverse and fascinating collection of essays on virtual heritage takes the 
reader on a tour of some of the most pertinent issues and challenges cur- 
rently facing the domain. There is a scholarly focus on the digital technologies 
deployed in a virtual heritage context, what these can achieve in terms of repre- 
sentation, and the relationship between precision, accuracy in representation, 
and the characterisation of authenticity. It is refreshing also to see considera- 
tion of less technical issues such as ethics and the related question of (digital) 
data preservation and long-term access. While these aspects of virtual heritage 
may be intimately related to various technical developments they also help to 
situate virtual heritage within the heritage domain more broadly by drawing 
out the similarities, and differences, between the ‘virtual’ and 'non-virtual in 
how the issues should be addressed. The impact of our interactions with the 
communities we serve and our obligations to care for and curate the content we 
create on their behalf are too often afterthoughts in project design. 

You will also find chapters here addressing key questions around visual effects, 
animation, visualisation, evaluation, and the role of gaming. Each topic is han- 
dled in a way that points to the importance of remembering human experience 
in all aspects of heritage. Heritage, especially virtual heritage, unquestionably 
benefits from the advanced technologies it can now mobilise. However, at the 
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heart of heritage there remains a dynamic and complex discourse around how 
the multiple values and diverse forms of significance attached to tangible and 
intangible heritage impact on communities in the present. 

The tools and methods deployed today under the banner of virtual heritage 
serve to inform and facilitate this discourse, rather than to stifle it with a spuri- 
ous authority derived solely from technical sophistication. A key task of the 
practitioner in virtual heritage is to recognise the active role that their work 
plays in informing heritage discourse and the impact that this discourse has on 
people, in the context of their personal and community identities, their politics, 
and even their livelihoods, through tourist and cultural economies. 

A fundamental way in which this role can be acknowledged is to break down 
the barriers between the professional and academic and the communities they 
impact. Much of the work of virtual heritage can be done with communities as 
well as for communities. If the ceaseless development of hardware and software 
tells us anything, it is that the next version, (of whatever it is), will be easier to 
use, and likely cheaper and quicker too. What appears to be the sole domain of 
the expert one day, for example, laser scanning, becomes a ubiquitous feature 
of a consumer phone handset the next. Consequently, there are few meaningful 
technical barriers to community co-production and engagement, that last for 
long (see Haukaas & Hodgetts 2016). What we do in virtual heritage and who 
we do it with and why, are bigger questions than what are the latest technolo- 
gies for recording or dissemination? 

It is part of the work of the virtual heritage practitioner to find and engage 
with the constituencies and communities that are invested in the heritage in 
question or to work together with these communities to develop alternative 
forms of significance around heritage that may be important to them, but 
which often lies out with the traditional regimes of significance operating 
in academia and cultural heritage management (see Jeffrey et al. 2020; Jones 
et al. 2017). 

It has been a common failing of much heritage work dealing with the distant 
past to assume there is no current community to engage with, and that work 
can be carried out as if the space in question is neutral or without stakehold- 
ers. This notion, itself now antiquated, is antithetical to the practice of heritage 
today and, in the virtual domain too, we must never take our eyes off the vari- 
ous communities with which, and for whom, we do our work. 

This collection of essays, with its broad mix of topics and perspectives, 
avoids the pitfall of treating its topic as an adventure in a technological play- 
ground. Taken together, the chapters and topics covered here provide the 
reader with a valuable springboard from which to explore virtual heritage. This 
includes technical developments and future directions, but also many of the 
broader questions around what we do in virtual heritage today and why we 
do it. 
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Virtual Heritage: From Archives 
to Joysticks 


Erik Malcolm Champion 


Curtin University; ANU; UWA 
erik.champion@curtin.edu.au 


While virtual heritage was initially described as a fusion of virtual reality (VR) 
technology with cultural heritage content (Addison 2001; Roussou 2002), as VR 
keeps changing, preservation of the content becomes increasingly problematic. 
Virtual heritage has been a (sometimes) successful communication medium but 
seldom has it succeeded as a preservation medium (Champion 2016). 

Even the term virtual reality has been used loosely, while the terms aug- 
mented reality and mixed reality may transform into more overarching terms 
like XR (extended reality), or merged reality. 

Stone and Takeo further emphasized the educational aspect (R. Stone & 
Ojika 2000): 


... the use of computer-based interactive technologies to record, pre- 
serve, or recreate artifacts, sites and actors of historic, artistic, religious, 
and cultural significance and to deliver the results openly to a global 
audience in such a way as to provide formative educational experiences 
through electronic manipulations of time and space. 


Ihave previously suggested that the purpose and significance of virtual heritage 
(Champion 2006) is more clearly defined as: 


the attempt to convey not just the appearance but also the meaning and 
significance of cultural artefacts and the associated social agency that 
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designed and used them, through the use of interactive and immersive 
digital media. 


If we follow the above definitions, then virtual heritage is not only the bringing 
together of virtual reality technology with cultural heritage content, it is also an 
experiential medium. However, that experience is based on the recreating or 
reconstructing of data, measurements, and observations. 

Virtual heritage is a fascinating and challenging area of practice and research, 
but papers seldom examine underlying assumptions and precepts or explain 
the complete design and testing process. Where academic work is published, it 
is typically behind paywalls. Where are the primers providing an overview of 
immersive technology applied to cultural heritage, directly and conveniently 
accessible to the public? And where does one learn to make the leap from meas- 
urement to user experience? 

Unlike other fields, virtual heritage projects are seldom long-lived and robust, 
clear and significant results are hard to find, data is seldom shared or easily 
accessible. Because of the many fields that help develop and present virtual her- 
itage projects and related technology, the lack of access to past projects and 
results, and the manic changes in related technologies, students and scholars 
from other fields face steep learning curves regards the technical opportuni- 
ties, the interaction design challenges, and the preservation risks. Our solution 
was to create a guide that is more concise, applied, and accessible to students 
from related fields. This is a key reason why this book is available as an open 
access publication. 


Measurement 


In the first chapter, Chapter 1: Speculating the Past: 3D Reconstruction in 
Archaeology, Robert Barratt explains the relationship between virtual heritage 
as discipline and as argument. To be an effective scholarly medium, virtual her- 
itage requires precision (because of its underlying computational nature), but 
also a way to convey the reasoning behind the measuring, the decision making 
informing the design of the simulations and conjectural models. 

Given that measurement is important, the interesting thing about meas- 
urement is how we learn from how we and others measure. The next chapter, 
by Hafizur Rahaman, Chapter 2: Photogrammetry: What, How, and Where, 
explains the increasing importance and versatility of photogrammetry. The time- 
consuming process of 3D model making can be replaced by photogrammetry, 
leaving us time to experiment with designing better user experiences. Interest- 
ingly, photogrammetry can work on personal devices or leverage supercomputers. 

From the camera arises the 3D model, but how do we give the 3D model life? 
Through representation and interaction, perhaps we could represent how these 
people understood and represented their world to each other. Animation and 
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modellings are key elements to producing captivating virtual environments, 
and there is now a range of impressive but free and open source software. In 
Chapter 3: Animating Past Worlds, William Carter explains how advances 
in animation as technology, discipline, and art can augment and improve the 
field of virtual heritage. 

However, there are two forms of measurement here: measurement of what 
has been left behind, and conveying the contextual measurement of what was 
built, created, and shared by the original peoples themselves - how did they 
measure and value elements of their culture? How did they map what 
they measured and why it was worth measuring? How could we communicate 
this to people today? The authors of Chapter 4: Mapping Ancient Heritage with 
Digital Tools explain how ancient concepts of place and journey can inform 
current initiatives to map past worlds. 


Interaction 


The above chapters have touched on interaction but are fundamentally con- 
cerned with visualisation based on measurement. The next group of essays 
focuses on the role of interaction in virtual heritage. Despite the promise and 
excitement of technological showcases, this is still a problematic component of 
virtual heritage. For instance, Roussou and Slater (2005) decried the current 
state of meaningful interaction and learning in virtual reality: 


Hence, the research question that emerges is how interactivity in a 
virtual learning environment can influence learning. 


Museums and galleries are ideal environments for interactive virtual heritage, 
but they seldom have the resources (Birchall & Ridge 2015) to fund full-time 
guided teaching and interactive-learning experiences; many may even lack 
space to display all oftheir collections (Bradley 2015). Hurt by the Coronavirus 
pandemic and drastic cuts in staffing and overall funding, they require robust, 
flexible, long-term technology. 

How do we address the challenges of museums and the wider GLAM sector, but 
also leverage their great virtues: as meeting places and as havens for education 
and edification? In Chapter 5: Hybrid Interactions in Museums: Why Material- 
ity Still Matters, Luigina Ciolfi explains how hybrid interaction can bridge the 
gap between technological showcases and the direct person-to-person virtues 
of the museum space. 

Perhaps we can also look to related industries for collaborative funding and 
development. For example, the video game industry is hugely profitable; Juni- 
per Research predicted that worldwide it would surpass 100 billion dollars in 
revenue in 2017 (Graham 2017). Commercial game studios can spend millions 
on capturing 3D digital assets and animating captivating environments. Their 
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allure and influence on virtual heritage is considerable, but their potential col- 
laboration is vastly more significant. 

In archaeology, there have been recent investigations of 'archaeogaming, 
defined as ‘the archaeology in and of video games’ (Aycock & Reinhard 2017). 
Designers are moving away from the principal goal of photo-realism, towards 
the potential of interpretation and conceptual learning (Roussou 2005). Gam- 
ing can be highly sophisticated or designed with primitive blocks by school 
children (such as Minecraft). In Chapter 6: Video Games as Concepts and Expe- 
riences of the Past, Aris Politopoulos and Angus Mol describe their teaching 
and research projects on archaeogaming and how this popular entertainment 
medium can also be educational. 

We might primarily consider virtual heritage to be only virtual reality, but 
mixed reality and augmented reality offer several advantages over virtual real- 
ity. Virtual reality (VR) is typically custom built and expensive for the general 
public, requiring specialist technology experts and facilities that are expensive, 
hard to source (Carrozzino & Bergamasco 2010), and difficult to maintain by 
specialist but cash-strapped heritage organizations such as galleries, libraries, 
museums (Ridge & Birchall 2015), let alone by the general public or share- 
holder communities. 

A possible solution is to employ mixed reality, augmented reality, or consumer- 
level virtual reality. In recent years, there have been increasing synergies 
between video games and virtual reality, thanks to increasingly powerful 
computers and the development of consumer-priced head-mounted displays 
(HMDs), see-through mixed reality HMDs, and smart-phone based augmented 
reality systems. In Chapter 7: Mixed Reality: A Bridge or a Fusion between Two 
Worlds? Mafkereseb Bekele explores whether mixed reality is an experience or 
a technology and how it can help bridge the past and the present. 

Interaction requires an audience, and mixed reality in virtual heritage, 
arguably, has not yet fully leveraged the engagement and value of participa- 
tive contexts. 


Ethics and Evaluation 


The last section outlines key issues in virtual heritage evaluation. However, 
before we set up our questionnaires and experimental designs, there are key 
ethical issues in cultural heritage and digital archaeology that need to be 
addressed but are seldom covered. In Chapter 8: Getting It Right and Getting It 
Wrong in Digital Archaeological Ethics, Meghan Dennis explains how technol- 
ogy is not morally neutral: there are successful projects that ‘get it right; but we 
can also learn from failed projects that 'get it wrong: 

Inadequate interactivity or a lack of reflection on related ethical and moral 
issues can lead to uncertain pedagogical benefits. In virtual heritage, arguably, 
evaluations are usually not conducted on the intended final audience, and the 
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experimental designs cannot be found or repeated by others. Although there 
are charters such as the London Charter and the Seville Charter, there are 
few publicly accessible models, and we lack a shared standardized evaluation 
dataset. In Chapter 9: Evaluation in Virtual Heritage, Panayiotis Koutsabasis 
explains the many evaluation methods available to virtual heritage, why it is 
a great field for user experience design, and criteria for ensuring a successful 
virtual heritage project. 

That is not to suggest virtual heritage has many examples exhibiting exem- 
plary usability or usefulness. Many researchers have complained about user 
experience issues and a scarcity of suitable pedagogical material (Karoulis, 
Sylaiou & White 2006) or the scarcity of data, collections and projects convey- 
ing the accuracy, authenticity and authorship of the simulated material (De Reu 
et al. 2012). 

Neither virtual heritage software nor commercial games are currently flexible 
enough to quickly and simply handle various content, nor do the models have 
real-time capability to link to scholarly publications and crowd-sourced mate- 
rial; hence many are stillborn (Guidazzoli et al. 2016). 

For example, (R. J. Stone, 2005) remarked: 


... VR delivered very little of use to the global IT community. A hand- 
ful of organisations actually adopted VR, but most were deterred from 
doing so by its complexity and cost. Today's VR supply companies have 
either passed away or are hanging on by a thread. 


There are too few successful and popular projects that people can find, access, 
and simply make work again (Economou & Pujol 2008). Many have simply 
disappeared or are only communicated via academic conference papers. For 
example, Thwaites (2013) recently warned that “We cannot afford to have our 
digital heritage disappearing faster than the real heritage or the sites it seeks to 
"preserve" otherwise all of our technological advances, creative interpretations, 
visualizations and efforts will have been in vain? 

There are major problems encountered when preserving cultural heritage 
in physical museums (Michaelis, Jung & Behr 2012), and the reliability of 3D 
data for long-term preservation is an on-going issue. However, one way of 
approaching the issues of preservation - and it is not yet a mainstream one - is 
to approach the challenges of preservation through rethinking the challenge 
of authenticity. If virtual heritage is based on digital capture and digital crea- 
tion, what is authentic? How are we trying to understand and preserve cultural 
authenticity when we employ digital technology? Why do so many projects in 
digital heritage avoid tackling the related issue of authenticity? And if they are 
not authentic (in some way), why do they deserve to be preserved? 

In the final chapter, Chapter 10: Authenticity and Preservation, I briefly 
try to explain this linked challenge and offer some potential short and long- 
term solutions. 
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Conclusion 


Virtual heritage is not yet a dependable and demonstratively effective com- 
munication medium; there are still too few examples of accessible useful and 
engaging models that one can test, verify, experience, and learn from. The field 
is in serious need of more contextual and useful usability research, but the big- 
gest issue is arguably preservation of the research data and 3D models. 
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Speculating the Past: 3D Reconstruction 
in Archaeology 


R. P. Barratt 
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rbarratt01 @qub.ac.uk 


Abstract 


This chapter introduces the main uses, methods, and issues of 3D approxima- 
tions. The practical advantages of using 3D approximations over traditional 
presentations methods is demonstrated, with a focus on realism, interactivity, 
and presence. Simple 2D images and enhancement such as gaming software 
offer multiple output formats for diverse aims. Additional uses, such as 3D 
simulations are also considered, demonstrating the use of these models for the 
interpretation of archaeological contexts. The chapter also contains a descrip- 
tion of standard methods of 3D approximating, using general guidelines 
applicable to a variety of software. 


Definitions and History 


In any field of scientific enquiry, presenting data in a simple and effective man- 
ner is essential for the propagation of information (Benko et al. 2004; Smith 
& Rosendale 1999). Especially in subjects with high public interest - such 
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as archaeology - theories, data and other results must be accessible both to 
experienced researchers and the general public. In the past 35 years, new digital 
technologies broadly labelled as visualisation have emerged. These new meth- 
odologies enhance archaeological presentation in traditional venues such as 
publications and exhibitions and introduce new digital knowledge repositories 
(Sifniotis 2012). 

The rise of visualisation techniques can be traced back to the mid-1980s, 
when the collaboration between archaeologists and computer scientists yielded 
the first 3D models of archaeological sites (for example Delooze & Wood 1991; 
Smith 1985). This movement was a result of recent developments in comput- 
ing, but contemporary changes in archaeological theory and practices also con- 
tributed to the adoption of digital techniques. 

Undoubtedly, visualisations path was paved by the processual movement, 
which advocated scientific enquiry in archaeological practices leading to 
new cross-disciplinary methodologies (Binford 1962; Clarke 1968; Willey & 
Phillips 1958). The development of GIS software in the 1980s, for example, 
demonstrated that computer science could be successfully used for archaeo- 
logical enquiry (Richards-Rissetto 2017). 

In recent years, the advancement of computing capabilities has led to pro- 
gressively more complex and diverse 3D models (Sifniotis 2012). The increased 
specialisation of visualisation methodologies has caused a shift from a single 
3D form, originally labelled virtual reality, to a wide range of different model- 
ling techniques. Visualisation can now be subdivided into a variety of methodo- 
logical groups, generally separated into survey-based and reconstruction-based 
techniques. Survey-based techniques include photogrammetry and laser scan- 
ning, which are discussed in Chapter 3, while the present chapter focuses on 
reconstruction-based visualisation. 

3D reconstructions! are user-generated virtual geometries primarily used for 
the presentation of real and hypothetical archaeological data (Figure 1). They 
are user-generated, as they require a modeller to manually input the geom- 
etry based on archaeological evidence and established theoretical elements. 
They are composed of a virtual geometry, created using xyz points, which are 
connected to form surfaces and solid objects. Unlike survey-based techniques, 
the subjects of the 3D reconstructions are both extant archaeological evidence 
and hypothetical elements based on established theories. 3D reconstructions 
are historically linked with the field of archaeological illustration and especially 
the work of Alan Sorrell, aiming to present an intact view of the archaeological 
past prior to destruction (Earl 2006; Georgopoulos 2014; Sorrell 1981). 

The main distinction between survey- and reconstruction-based techniques 
lies in their use. Survey-based techniques help preserve material evidence by 
creating a permanent digital copy, especially of features uncovered during exca- 
vation (for example, Olson et al. 2013). 3D reconstructions are not faithful ren- 
ders of reality, but they have the capability of synthesising theory and evidence 
into an accessible medium. As such, 3D reconstructions can present hypotheses 
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Figure 1: A 3D reconstruction of a Neolithic hut in Malta. The model has been 
used in publications and exhibitions to show the original form of the struc- 
ture prior to destruction. 


dynamically and intuitively to the public, as well as occasionally provide an 
environment for experimentation. 


The Reconstruction Process and Uses 


3D computer models are composed of a series of points in virtual space, which 
are connected to form textured triangular surfaces. In 3D reconstructions, 
these points are inputted manually by the researcher using tools provided by 
the 3D software. Common modelling programs such as SketchUp, Blender, and 
3ds Max allow users to create simple shapes or surfaces that can be manipulated 
through pulling or extruding tools to generate more complex elements. More 
recently, procedural generation allows the creation of large-scale models such 
as cityscapes through a rule-based methodology (Adao et al. 2012). However, a 
3D reconstruction is primarily an artistic process controlled by the modeller. 

3D reconstruction starts with the archaeological data, and surviving features 
are recreated in the model using plans, sections, and measurements obtained 
during excavation. These provide a realistic basis for the model. However, 
archaeological data is limited, and elements that are no longer in situ must 
be created by the modeller based on limited sources, often by actively choos- 
ing from several plausible scenarios. Dell'Unto et al. (2003) identify a range 
of references that aid the reconstruction process, from photographs, literature, 
similarity with other parts of the site, comparison with different archaeological 
contexts and replication of the style of the period. 
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The reconstruction process is therefore an investigative methodology, similar 
to archaeological theory building where a variety of sources are connected to 
identify and support a hypothesis. The reconstruction process can itself provide 
new insight into archaeological interpretations (Barceló 1992). By viewing data 
in 3D space, relationships between individual parts are visualised, often leading 
to observations that were not apparent in the 2D data (Lulof et al. 2013). 

3D reconstruction’s ability to visualise the relationship between elements 
has made it a valuable tool for the presentation of archaeological contexts 
to the public. Archaeological remains are often partial, requiring visitors to 
imagine missing elements despite lacking the necessary expertise. How- 
ever, 3D reconstructions provide intuitive and immediate access to complete 
archaeological contexts. 

Until recently, 2D still images were the main form of presentation for 3D 
models, but in recent years the development of serious games — video games 
designed for educational purposes - has changed the way archaeological sites 
are showcased (Anderson et al. 2010). Serious games use gaming engines such 
as Unity3D or Unreal Engine to create complex virtual worlds that provide 
an engaging and stimulating experience to the user (Figure 2). Archaeological 
sites are fully recreated in 3D space that the user can explore in the first-person 
perspective, often directly interacting with the virtual environment. 

This new generation of 3D reconstructions exploits video games' ability to 
create presence to facilitate learning. Presence is the feeling of belonging in a 
digital environment without awareness of mediation (Biocca & Levy 1995). As 
the environment creates responses that are analogous to real-life stimuli, the 
user believes this digital world to be an extension of reality. The virtual envi- 
ronment possesses inbuilt characterises that mimic natural response (i.e., the 
ability to walk through water or the capacity of a wall to block movement) 
and creates a strong sense of belonging by allowing meaningful interactions 
between the user and the space (Pujol & Champion 2012). The characteristics 
that create natural responses are comparable to Gibson's affordances, which in 
turn are inspired by Heidegger's thrownness (Gibson 1979; Heidegger 1927). 

The feeling of presence is unique to serious games and presents advantages to 
traditional methods of teaching. Presence has been closely linked with learn- 
ing, as users subconsciously acquire knowledge through the engaged explo- 
ration of virtual space (Herrington et al. 2007; Lacasa et al. 2008; Rosenberg 
2006). Serious games are associated with constructivism, which is 


... [the] view that learners assimilate knowledge by engaging in self- 
directed learning activities that are accomplished through constructive 
tasks (Roussou et al. 1999: 250). 


By actively engaging with virtual environments the user gains an understanding of 
the past driven by curiosity and individual choice. Nonetheless, the learning 
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Figure 2: A serious game that allows exploration of a burial site in Malta. The 
interface provides tools for self-guided learning. [Author]. 


process can be guided through the use of narration, characters, and quests, thus 
providing an environment rich in knowledge without the constraints of tradi- 
tional methods of dissemination (for example, Champion et al. 2012). 

Elements such as interaction, embodiment, and realism contribute to the 
sensation of presence. Interaction is achieved by creating opportunities in 
which the user can test the world, by engaging with virtual objects and meas- 
uring the naturalness of the responses (Roussou 2004). Embodiment is the 
sensation that the character controlled in the serious game is an extension of 
the user's physical self (Biocca 1997). This feeling helps the user to experience 
stimuli directly, without a sense of mediation. Realism is the closeness of the 
virtual spaces to reality, which is created through complex geometry, textures, 
and shading (Gillings 2001). Additional elements such as social presence, cul- 
tural presence, immersion, stimulation of other senses, and relation to others 
play a role in generating presence. 

The use of 3D reconstruction is not limited to presentation. The flexibility of 
the models and the ability of the user to manipulate the geometry allows for 
archaeological experimentation. 3D reconstructions can be used to test theo- 
ries, by creating different scenarios and observing the relationship between ele- 
ments (Barceló 2001). In such cases, the 3D reconstruction acts as a simulation 
- a scientific experiment conducted in a digital environment where a system 
replicates reality (Lake 2014). 

3D reconstructions can represent digital proxies of archaeological contexts 
within which hypotheses are tested. Physics simulations can, for example, be 
used to verify the stability of structures based on the available archaeological 
evidence (Levy & Dawson 2006). Crowd simulations can show how architec- 
tural features affect the movement of people within the site (Maim et al.. 2007). 
Astronomical studies provide new information regarding the construction and 
use of structures (Frischer & Fillwalk 2012, 2013). Therefore, 3D reconstructions 
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offer the possibility of exploring queries and theories that are otherwise inac- 
cessible with traditional archaeological practices. 


Hyperrealism, Uncertainty, and Possible Solutions 


Despite the advantages of using 3D reconstructions for the presentation and 
interpretation of archaeological data, these methodologies have encountered 
resistancefrom thewiderarchaeological community (McCoy &Ladefoged 2009). 
The main criticism raised in the literature is 3D reconstructions tendency 
to mislead the end-user through highly realistic imagery (Eiteljorg 2000). 
Although hypothetical elements are mostly based on archaeological evi- 
dence, in most venues of publication the 3D reconstructions are stripped of 
archaeological sources. These 3D reconstructions often lack the required 
tools to identify their overall accuracy, while simultaneously carrying a sense 
of truth. Borrowing from the concept of hyperreality introduced by Baudrillard, 
the reconstructions are an intermingling of real and hypothetical without any 
means of distinguishing the two (Baudrillard 1983, 1988; Forte 2011). At the 
same time, images carry a sense of realism and legitimacy that leads users to 
automatically believe them to be truthful, similar to Benjamin’s (1936) aura 
in art or as represented in Magrittes (1928-29) The Treachery of Images. As a 
result, 3D reconstructions carry a risk of presenting as fact a hypothesis with 
little supporting archaeological evidence. 

Several researchers have attempted to provide visual cues within 3D recon- 
structions to address their inaccuracy (Dell'Unto et al. 2013; De Luca et al. 
2014; Georgopoulos 2014; amongst others). Colour codes, wireframes, or point 
clouds are included within the 3D models to show the precision of individual 
parts, akin to the pink cement used when physically reconstructing archaeo- 
logical sites. The non-photorealistic movement initiated by Strothotte et al. 
(1999; Masuch & Strothotte 1998; Masuch et al. 1999) suggests using different 
rendering styles to highlight uncertainty. 

On the other side of the argument, researchers have argued that 3D recon- 
structions should not compromise realism for the sake of accuracy. Gillings 
(2001) suggests that the main focus of 3D reconstructions should be perceived 
accuracy, proposing that a model should provide a realistic experience regard- 
less of imperfections. He quotes Dovey's (1985) claims that an artificial beach 
in the desert may not be a physical substitute for a real beach, but still provides 
the same experience. 

While the issue of misrepresentation in 3D reconstructions is still under dis- 
cussion, in recent years several documents such as the Seville Principles and the 
London Charter have been published (Denard 2012; Seville Principles 2011). 
These documents present a theoretical framework for 3D reconstruction, in 
order to legitimise the field through a formalised and standardised method- 
ology. By ensuring 3D reconstruction follows accepted guidelines, the risk of 
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misinterpretation can be assessed and minimised. Amongst other observa- 
tions, these documents advocate for the recording of uncertainty in 3D recon- 
structions through the use of metadata and paradata. Metadata and paradata 
store necessary information regarding the reconstruction process, in the form 
of software data (metadata) and records of the decisions taken by the modeller 
(paradata) (D'Andrea & Fernie 2013; Denard 2012). By making this data acces- 
sible, it is possible to determine the overall reliability of the 3D reconstruction 
and identify hypothetical elements that require further consideration, allowing 
it to be open and replicable. 

However, despite these conscientious approaches, more work is necessary. At 
present, metadata and paradata are not a requirement for 3D reconstruction 
projects, and even when these data are recorded, issues such as a lack of online 
repositories and non-standardised datasets impede their proliferation (Ince 
et al. 2012). 


Conclusions: Looking to the Future 


3D reconstruction is a relatively new field in archaeology and as such it has both 
untapped potential and unresolved issues. Serious games offer new opportuni- 
ties for public engagement, by providing a learning experience driven by curi- 
osity in an interactive medium. Further collaboration with the field of video 
game development could provide more powerful and focused 3D reconstruc- 
tions. By fully embracing the techniques that stimulate the users in games and 
by creating a stronger sense of presence, archaeological information can be 
presented more effectively. For the interpretation of archaeological sites, the 
similarities between simulations and 3D environments demonstrate the pos- 
sibility of using 3D reconstructions for complex experiments that can further 
archaeological knowledge despite limited evidence. 

Yet, despite the potential, more work is needed in the development of a sat- 
isfactory theoretical background to 3D reconstruction. The issue of inaccuracy 
and the possibility of misleading the public through erroneous hypotheses 
require further consideration. The use of metadata and paradata is promising as 
it provides accessibility to the reconstruction process, but changes in publishing 
techniques are necessary to accommodate the recording of new information. 
Overall, 3D reconstructions can change archaeology for the better, but only by 
accepting their limitations and by ensuring scientific rigour is maintained. 
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Notes 


! Reconstruction-based models have been called a variety of names, includ- 
ing 3D reconstructions, virtual reality and 3D approximations. Although 
the term 3D reconstruction has limitations, as described in Clark (2010) 
and Barratt (2020), it is adopted in this context as it is the most common; 
3D model is also employed in the text. 
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Abstract 


Developing 3D digital models of artefacts, monuments, excavations, and his- 
toric landscapes as part of digital documentation is becoming commonplace 
in the fields of heritage management, virtual tourism, immersive visualisation, 
and scientific research. Such 3D reconstruction or 3D data acquisition from 
a laser scanning process involves high costs, manual labour, and substantial 
expertise. On the other hand, image-based 3D modelling photogrammetry 
software offers a comparatively inexpensive alternative and can handle the task 
with ease. Besides, documenting heritage artefacts with free and open-source 
software (FOSS) in supporting photogrammetry is getting popular for quality 
data production. 

Due to the present pandemic situation and social distancing restrictions, gal- 
lery, library, archive, and museum (GLAM) industries are facing an incremental 
burden on both their income and visitor traffic, which is affecting their sur- 
vival. Asa way out, we can see some GLAM institutes are trying to expand their 
collections on digital platforms for showcasing and promoting virtual visits. 
Numerous online portals and repositories are evolving for archiving, sharing, 
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and trading 3D models are also evolving to support this digital ‘vibe. This 
chapter explains the basics of photogrammetry and its development workflow, 
including data acquisition (photo shooting), data processing, and a few post- 
processing tools. 


Introduction 


In areas such as heritage documentation, virtual recreation, simulation, crime 
scene analysis, urban project planning, augmented/virtual/mixed reality, 
serious games, and scientific research, 3D digital models of artefacts, build- 
ings, archaeological excavation, and historic natural landscapes are becoming 
increasingly popular. Applications such as Maya, Blender, or 3D Studio Max, 
which follow traditional geometry-based modelling methods, involve a steep 
learning curve and require significant time and energy. However, advances in 
hardware (laser scanners, unmanned aerial vehicles, etc.) and software, have 
increased the opportunities for the virtual reconstruction of 3D scenes. 

Lidar, also called 3D laser scanning, measures distances (ranging) by illumi- 
nating the target with laser light and measuring the reflection with a sensor. 
One can often refer to this pathway as the most suitable for providing the most 
accurate 3D models (depending on the environment and the 3D scanner). 
However, laser scanners and structured lighting systems are often expensive 
and require a high level of expertise. Furthermore, this technology has limi- 
tations regarding rendered material properties and environmental conditions, 
for example, dealing with strong sunlight (Nguyen et al. 2012). 

There is an urgent need for tools and supports that enable non-expert users 
to build 3D reconstruction models comfortably and efficiently, especially for 
3D digital documentation and heritage visualisation purpose. In supporting 
this demand, commercial software packages as well as free and open-source 
software (FOSS) based on image-based modelling (IBM) or photogrammetry 
have emerged. Regard3D, Colmap, VisualSfm, and Python Photogram- 
metry Toolbox are some leading examples of FOSS. 

The image-based 3D modelling/reconstruction method uses uncalibrated 
photographs from cameras and can generate a 3D point cloud via various algo- 
rithms. A certain number of quality photographs, therefore, are required to 
allow the software to process, match, and triangulate visual features and to fur- 
ther generate the 3D point-cloud data. Structure from motion (SfM) is one of 
the most popular techniques in the image-based modelling method that has 
been employed in numerous software packages. This procedure helps non- 
expert users to capture high-quality models rapidly and conveniently from 
uncalibrated images taken with an ordinary camera, without any advanced 
equipment or specially designed lighting conditions. 

A phone camera and a personal computer with a moderate graphics card 
are good enough to process 3D models of small objects and artefacts from 
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photographs. Nevertheless, digitising large-scale buildings or landscapes with 
thousands of photos may require significant hardware support and processing 
time. If the user has limited hardware, has a weaker graphics card (CUDA capable 
NVIDIA video card), or is working with large data sets, he/she might need to seek 
the help of cloud computing. Examples of commercial services supporting large 
data processing (including photo captured from drones/UAVs suitable for archi- 
tecture, engineering, and construction [AEC] and survey industries) include: 


e Autodesk ReCap Pro (https://www.autodesk.com/products/recap/) 
e Get3d (www.get3d.cn) 
e Pix4Dcloud (https://www.pix4d.com/product/pix4dcloud) 


There is also non-commercial free cloud processing support for remote 3D 
reconstruction, such as ARC 3D Webservice (https://www.arc3d.be/). For 
cloud-based processing, users can upload photos/data set through a client tool 
and get informed by an email sent automatically from the server when the pro- 
cess is done, and the model is ready to download. An experienced user with 
some technical expertise can also ‘rent’ virtual machines (called cloud comput- 
ing instances or instances) from commercial services like Amazon Web Service 
(AWS), Microsoft Azure, and Google Cloud for having more customised and 
robust performance. These virtual machines have the necessary power to pro- 
cess large photogrammetry projects. However, these commercial services come 
with various subscription model tiers. 

Smartphones can be used for scanning and extrapolating a 3D model from 
an existing object with the aid of applications. These apps use a phone camera 
to scan an object and can process the data either within the phone or can send 
the data to an online server for remote processing. Examples of such applica- 
tions include Qlone, Scandy Pro, Scann3D, Capture:3D Scan Anything, Trino, 
and Sony 3D Creator. Models generated from such an app may be less accurate 
than models derived from 3D scanners. Still, it is a great and affordable way to 
learn about 3D scanning without investing in high-end hardware and software. 

There are some other hardware-based solutions for 3D scanning and model- 
ling, such as Occipital Structure Sensor (https://structure.io/) connected via an 
iPad. A specific app allows 3D scanning, processing, and exporting (via email) 
the 3D model to specific formats and locations to support the device. The soft- 
ware is free to use. 

We can also see some other emerging technologies and recent hardware 
development, assisting 3D scanning tasks. For example, the recent inclusion of a 
lidar scanner on the rear camera of the iPad Pro/iPhone 12 Pro looks promising 
and may help in mapping the environment and related 3D objects. Apples lidar 
scanner is not yet at the level of professional lidar scanners used for outdoor 
surveying and scanning. However, it can still measure the distance to surround- 
ing objects up to approximately five meters away, and it works both indoors 
and outdoors, at a photon level of nanoseconds. It can also track the location of 
people (spatial tracking) (Narain, 2020). A portable 3D laser scanner (such as 
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FARO Freestyle 2) can also support fast and photorealistic 3D capture. The 
lightweight handheld device has a high degree of flexibility, which allows its 
operator to scan various places with challenging accessibility and conditions. 

In general, the 3D reconstruction process comprises two parts: data acqui- 
sition and data processing. Data acquisition can be made through various 
hardware ranging from mobile phone cameras to satellite imaging. The data 
processing/3D reconstruction, on the other hand, can be undertaken with a 
mobile device, a local PC, or with the support of cloud-based computation. 
However, the selection of the hardware and software largely depends on the 
size and location of the object, allocated budget, technical expertise, and access 
to the service. 

Image-based 3D modelling technique can be executed either in a local com- 
puter or in a remote computer/cloud server based on the adopted software/ser- 
vice. Software or services such as ARC3D and Autodesk Remake use the power 
of cloud computing to carry out the data processing in a remote server. By con- 
trast, software including Regard3D, Aspect3D, Metashape, 3D SOM Pro, 3DF 
Zephyr, and so on process the data on a local (client) machine. This chapter does 
not cover the cloud processing method; instead, it presents a workflow solely 
featuring software/applications that run on a local computer or workstation. 


Photogrammetry 


Photogrammetry is all about ‘measurement’ of the subject of the image. A pho- 
tographer needs to follow a specific rule-based procedure to collect the data set 
(photographs) to get a high-quality photogrammetric measurement. This stand- 
ard photographing procedure, therefore, guides the photographer about config- 
uring the camera, selection of certain objects, shooting techniques (overlapping), 
and positioning and orientation of the camera towards the subject in such a way 
that the processing software can minimise the uncertainty and produce the best 
result. A 3D model (or point clouds) can only be achieved when the relations 
between photographs are appropriately established. The success of such photo- 
grammetry-based 3D modelling largely depends on the photographic data and 
the underlying algorithm of the processing and measuring software. 

This 3D documentation is of prime importance in the fields of historic pres- 
ervation, tourism, educational, and real state (Bertocci, Arrighetti & Bigongiari 
2019; Dhonjua et al. 2017). Image-based reconstruction or photogrammetry 
process claims to be cost-effective as compared to traditional laser scanning 
methods and can provide an automated system with considerable accuracy in 
the 3D model generation (Scianna & La Guardia 2019; Skarlatos & Kiparissi 
2012). However, there are expenses to be incurred in acquiring commercial soft- 
ware licenses, and a certain level of technical skill and knowledge is essential. 

Several articles have described the overall production of 3D models as a 
sequence of calibrated or uncalibrated photographs (Fuhrmann, Langguth 
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& Goesele 2014; Schóning & Heidemann 2015; Skarlatos & Kiparissi 2012), 
including details of different techniques for achieving a high degree of accuracy 
(Bolognesi et al. 2014; Nguyen & Dang 2017). 

One can also find comprehensive studies on image-based modelling 
software that analyses their performance (Durand, Engberg, & Pope 2011; 
Grussenmeyer & Al Khalil 2008; Scianna & La Guardia 2019; Wang 2011), accu- 
racy in 3D production (Bolognesi et al. 2014; Deseilligny, Luca & Remondino 
2011; Oniga, Chirilá & Státescu 2017), algorithms used (Knapitsch et al. 2017), 
and scalability (Knapitsch et al. 2017; Santagati, Inzerillo & Di Paola 2013). A 
few studies also address workflows as well (Hafeez et al. 2016; Koutsoudis et al. 
2008). However, research on either best practice or automated workflows that 
can be adopted by the general public to create free and easy 3D reconstruction 
models are hard to find, and low-cost workflows are scarce. 


Photogrammetry-Based 3D Modelling Workflow 
Photography/Data Collection 


Photographs can be acquired from several cameras or a single camera. Uncali- 
brated photographs taken from general cameras are usually used to create a 3D 
point cloud by image-based 3D reconstruction software. The software deter- 
mines the geometric properties of the objects based on the provided photo- 
graphs and some specific properties of the camera, such as the focal length and 
sensor size. This process requires comparing and referencing points or match- 
ing pixels across a series of photographs. Indeed, the quality and number of 
photographs play a vital role in allowing the algorithm to process the surface, 
match points, and triangulate the visual features. A successful compilation of 
the previous steps can then generate the 3D point cloud. 

The camera can range from a mobile phone camera to a professional DSLR 
camera for photo shooting. Photos must be taken with the right amount of 
overlap while repositioning the camera for every shot. Depth of field is recom- 
mended as deep as possible (E8 or higher) so more objects are in focus. The 
photos should not be over- or underexposed, as dark shadows and washed-out 
lighting can mask important details when processed. Shadows on an object 
may be mistaken by the photogrammetry software and can produce holes in 
the object, which will eventually be reflected in the resulting reconstruction. 
For best results, diffused light is preferred to avoid overexposure. 

Background also plays a vital role in the output quality of the 3D model. 
The reconstruction may appear wrapped if the background prevents the object 
from being identified or distinguished from the surfaces or objects behind the 
subject. Objects with a glossy surface, uniform colour or texture, and trans- 
parent or translucent material will not yield a quality reconstruction. These 
objects generally produce holes in the resulting meshes or extensive warping. A 
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mesh is a discrete representation of a geometric model in terms of its geometry, 
typology, and associated attributes (Comes, Buna & Badiu 2014). Therefore, it 
is often recommended to avoid dark surfaces, reflective surfaces, transparent 
surfaces (including water), uniform textures and solid colour surfaces, moving 
light sources and shadows, and capturing the photographer's own shadow. 

Generally, the object of interest should take up 7096 of the frame or more. 
Photographs should overlap every 5-10 degrees while capturing an object. The 
rule of thumb is, "Ihe more your photos overlap, the better your reconstruc- 
tion: Therefore, at least 5096 of a photo needs to overlap with another. The more 
photos are captured, the greater chance of having an accurate model. For 
more information about image acquisition, please see Lab (2018). 


Data Processing/3D Model Development 


Structure from motion (SfM) is one of the most popular methods and often 
been recommended by many authors for 3D reconstruction (Nikolov & Madsen 
2016). Ranging from simple home-brew systems to high-end professional 
packages, there are a wide variety of 3D modelling programs available based on 
SfM in the market. Rahman and Champion (2019) present a comparison study 
of four popular free and open-source software (FOSS) packages that use the 
SfM techniques. This study, however, generalises the photogrammetric work- 
flow as a six-step process to produce 3D reconstructions and 3D models, which 
includes the following: 


(1) Add photos (or image acquisition), 

(2) Align photos (or feature detection, matching, triangulation), 

(3) Point cloud generation (or sparse reconstruction, bundle adjustment), 

(4) Dense cloud generation (or dense correspondence matching), 

(5) Mesh/surface generation, and 

(6) Texture generation (a few software packages also offer cloud/mesh edit- 
ing, such as Metashape). 


Data/photo sets must be imported to the software first. Usually, users are given 
options to add/import the photos all at once or as a group (or chunk). The soft- 
ware analyses each photo through an algorithm (such as A-KAZE) and detects 
features (sometimes also called key points). ‘Features’ are points in an object 
that have a high chance of being found in different images of the same 
object, for example, edges, corners, marks, and the like. 

For each feature, a mathematical descriptor is calculated. This descriptor 
has the characteristic that can identify the same point in an object in different 
images (seen from different viewpoints). Techniques such as LIOP (local inten- 
sity order pattern) can be used for this purpose. The descriptors from different 
images are then matched and geometrically filtered. This process results in a 
collection of matches between each image pair. 
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Figure 3: A typical workflow of photogrammetry data processing 
(3D reconstruction). 


The next is the calculation of the ‘tracks? Each feature that is part of a match 
in an image pair is also searched for in other images. A track is then gener- 
ated from features if these features satisfy certain conditions, for example, if a 
track is seen in at least three images. Then it goes to the ‘triangulation’ phase. 
Matches from all the image pairs are then used to calculate the 3D position and 
characteristic of the ‘camera’ In this enormous process, the algorithm finds the 
location where each image was shot, including the visual characteristics of 
the camera, followed by calculating the 3D position of each track. 

The triangulation phase produces a sparse point cloud. To obtain a more 
dense point cloud or ‘densification, several other algorithms are used. Finally, 
it comes to surface generation. The point clouds are used to generate a surface, 
either with coloured vertices or with texture. However, not all photogrammetry 
software offers mesh editing, mesh optimising, cap holes, and texture genera- 
tion. For a better result, additional software, with some additional steps, may 
need to be used. 


Further Improvement/Mesh Generation and Editing 


Photogrammetry software usually generates noise, outliers, and irrelevant 
points while developing the point clouds, and the result requires cleaning. 
Not all packages support editing and cleaning. The creation of a mesh from 
uncleaned point clouds is not recommended, as it will require more time for 
computation and eventually will generate unwanted surfaces. Once the mesh 
is developed from the point clouds, software packages such as 3D Studio Max, 
Maya, ZBrush, Blender, and MeshLab can be used for further processing. 
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Figure 4: Original versus 3D reconstruction of objects. 


The generated point clouds can be imported directly to MeshLab (http: 
//meshlab.net), which is free and open-source software (FOSS). MeshLab pro- 
vides various tools for selection and removal of points/vertexes (noise, cleaning 
outliers), surface reconstruction (or mesh generation), simplification of mesh, 
and ‘cap hole’ (or close holes tools closes void areas on the model substituting 
photogrammetric reconstruction with extrapolation data) steps that enhance 
the 3D model. 

Mesh simplification is self-explanatory. It decreases the number of poly- 
gons while keeping the shape as close as possible to the original. Because of 
the reduction of the poly count, processing is faster in 3D visualisation or in a 
game engine. Cap holes close or cover any gap or hole where the previous mesh 
generation failed to generate a polygon. 

‘Texturing provides visual skin/membrane coverage for the 3D models, so 
that the virtual objects resemble the original and look realistic. MeshLab can 
export a wide range of 3D file formats with supporting textures (e.g., *.x3d, 
* obj), including 3D point cloud with vertex or points colour (*.ply). 


Applications and Possibilities 


The application of photogrammetry covers the broad areas of topographic 
mapping, architecture, engineering, manufacturing, quality control, police 
investigation, cultural heritage, geology, and the film and game industries. 
Virtual heritage, digital preservation, and archiving: Archaeology and cultural 
heritage both use photogrammetry for digital preservation of heritage assets 
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Figure 5: Dense point cloud generated from photographs (Sompur Mahavihara, 
Bangladesh, a world heritage site) using photogrammetry. 


and sites. There is a remarkable number of public, commercial, and hobbyist 3D 
repositories for hosting and archiving 3D heritage models. CARARE, Europeana, 
Smithsonian, TurboSquid, Sketchfab, and the like support hosting and archiv- 
ing 3D heritage assets. There are more than fifty commercial repositories, such 
as ShareCG, Turbo Squid, Blendswap, and MyMiniFactory. However, they are 
mostly commercial marketplaces and do not necessarily support 3D archiving. 

Land surveying: Photogrammetry and land surveying have many similarities, 
and that is why this technique has become popular in this field. Besides satel- 
lite imagery, drones are getting popular as a cheaper alternative for obtaining 
images of contours and land masses. Land surveyors and building professionals 
can utilise photogrammetry for automated generation of 3D surface models, 
developing rectified and orthorectified imagery and localised mapping from 
photography gathered by a UAV (drone). 

Forensic and accident reconstruction: Photogrammetry has been used suc- 
cessfully for crime scene investigation, forensics, and traffic accident recon- 
struction and measurement. The technology is equally applicable for both 
indoors and outdoors. Recording tiny details are essential in these cases, and 
photogrammetry can help in a court of law by documenting the precise meas- 
urements of the crime scene. 

Filmmaking: Filmmakers have relied on 3D modelling and 3D animations for 
decades. Photogrammetry is now supporting this demand by providing a more 
realistic and accurate 3D environment. Close-range digital photogrammetry 
can be used in developing accurate 3D movie sets, while the location scout- 
ing can be done virtually using aerial photogrammetry. Photogrammetry can 
also be used in filmmaking for creating digital stunt doubles, texture mapping, 
crowd replication, and more. 
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3D assets and environments for games: Photogrammetry is being used in 
video games to create high-quality 3D assets and 3D realistic environments. 
The developers of "The Vanishing of Ethan Carter" first applied this technology 
in 2014 for creating stunning settings for the game. Since then, photogramme- 
try has been extensively used for 3D. 

Building design and renovation: Building renovation requires drawings as a 
reference point for decision making for architects and associated professionals. 
However, construction drawings are often rarely found for historic buildings. 
In these cases, photogrammetry software can be used to create point cloud and 
3D models, which can later be converted to CAD drawings and BIM (Building 
Information Modelling). Photogrammetry software is equally useful in survey- 
ing the surrounding area for further use in landscape design. 

Geology and mining: Disciplines such as geology and mining use photogram- 
metry for various applications. Photogrammetry provides affordable solutions 
in generating point clouds and elevation maps, which can be used as compre- 
hensive analytical tools to measure stockpiles and pits instantly. 

3D printing and prototyping: Industries including the automobile, jewellery, 
and medical training industries use photogrammetry and 3D printing. Instead 
of creating an object manually, photogrammetry can transform images into 3D 
models and can cut down the required time in model production. Once the 
model is captured from the real-world object, it can be reproduced at any scale 
(Lima et al. 2019). 

Photogrammetry in sports: Close-range photogrammetry is used for evalu- 
ation of players performance and in developing virtual training systems. 
A photogrammetric system can track and measure the most accurate body 
movement of an expert player such as a javelin thrower's shooting motion. 
These parameters then can be used to train new players. Analysing payers 
movement (biomechanical differences) also allows the scientists and athletes 
to minimise adverse body effects, such as an improper movement of the 
elbow joint during the slap-shot of ice hockey athletes (Caniberk, Sesli & 
Getin 2017). 


Conclusion 


Due to the recent concerns regarding the destruction and damage inflicted 
on internationally recognised heritage sites in Afghanistan, Syria, Iraq, and 
most recently in Brazil and France, digital documentation and 3D preser- 
vation of historical and monuments have progressively entered the arena of 
international importance. Thanks to emerging computer technologies and 
the advancements in photogrammetry software, which is helping us to stream- 
line the process of 3D documentation and digital conservation. In particular, 
the development of affordable techniques such as image-based photo modelling 
and free and open-source software (FOSS) is undeniable. The rapid adoption 
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and application of photogrammetry in domains other than architecture and 
archaeology is also remarkable. 

The process of image-based modelling is easy; however, photo shooting for 
this purpose requires specific skills and following certain rules. The software 
can only produce the 3D model (or part of it) if it is visible to the camera. This 
means any unseen section in the photograph will produce holes in the model. 
Not all objects are suitable for image-based photo modelling. More photos can 
produce better 3D models, but at the same time, more data sets require more 
powerful hardware and graphic card support. Free and open-source software 
is not always bug-free, and the developer rarely provides technical support. 
Regardless of these limitations, photogrammetry or image-based 3D modelling 
is a preferable solution for digital documentation and preservation of heritage 
and cultural artefacts, buildings, and sites. 
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Abstract 


Creating ‘past worlds’ is more than just creative flair or technical wizardry, it is 
the distillation of grounded heritage interpretation and reflection as applied to 
the creative (re)visualization of ancient peoples and landscapes. Be it the digital 
dinosaurs of Jurassic Park or the fake placement of lifelike digital actors seam- 
lessly inserted within media, virtual heritage is increasingly caught between 
the praxis of the visual enchantment of feature film and television visual effects 
(VFX), the increasing visual and phenomenological immersive worlds of 3D, 
virtual games and the hyper-reality of deep fake VFX. This chapter will provide 
the basics of animation, while at the same time introducing the reader to the 
concepts of virtual archaeology and digital cultural heritage from a digital visu- 
alization perspective. 


Introduction 


The concept of animation has been around for centuries and archaeologi- 
cally, I would argue, reaches further back into hominid evolution. The walls of 
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Figure 6: Adventure Man (2020) by Prof. Kris Howald, Sheridan College 
Computer Animation Program - Example of Animated Keyframes in 
Autodesk Maya. 


Chauvet Caves, with their 37,000-year-old brilliantly sketched and painted 
depictions of hunting, prehistoric animals, and mythical creatures, dance when 
lit by fire or torchlight, as if animated in real time (see Azéma & Rivére 2012). 
Historically, the zoetrope, referred to as the ‘wheel of life’ is probably the most 
iconic tool or projector of animated images (Krasner 2013: 3). Invented by 
William George Horner in 1834, it included a cylinder with vertical slits on 
a base that allowed the cylinder to spin. When the cylinder was spun and the 
viewer looked through the slits to see a sequence of still images, animation 
could be perceived due to the rate of speed and the viewing angle. 

However, it was the ground-breaking work of Victorian era photographer 
Eadweard Muybridge, who invented photographic technology to capture the 
movement of humans and animals, that formed the basis of animated film (see 
Baker 2007; Fresko 2013) working in Leipzig who made the next major contri- 
bution based on very simple measurements. In 1912, Windsor McCay took the 
concept of animated film and created one of the first animated shorts, Gertie 
the Dinosaur, inventing the use of keyframe animation and registration marks, 
which truly launched the artistic and technical process of animating images or 
objects (see Callahan 1988). 

Today, there is an unprecedented use of animation, not only in entertain- 
ment but also in advertising, scientific visualization, and political propaganda. 
Animation is a convincing means to visualize imaginary characters, worlds, 
and concepts; combined with hyper-real image-rendering techniques, it can 
compel the viewer's mind to think that what is being seen could in fact be real. 
Although this chapter will not unpack the very real concern of how anima- 
tion is now being used to promote alternative truths (see Westerlund 2019), it 
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is important for us, as digital cultural heritage (DCH) practitioners, to reflect 
on how our unconscious biases influence the animated images we create and 
how those images also greatly influence how audiences engage with the virtual 
ancient cultural heritage worlds we illuminate. 

We will explore the basics of ‘animation’ per se and how certain techniques 
can be used to enhance your digital cultural heritage research and potential 
skill-set. Given the huge range of animation techniques in various forms, such 
as 2D, 3D, real-time and stop-motion, I will focus on the use of 3D in film, 
virtual reality, and interactive gaming, which has recently exploded into the 
academic realm of digital cultural heritage. 


What Is Animation? 


The concept and word animate or animato comes from the ancient Latin 
meaning ‘to give life’ (Lewis & Short 1879). The current industry term ani- 
mation also refers collectively to the creative and technical process in which 
to animate, whether it be 2D, 3D, or gaming environments. Animation comes 
in many forms: the traditional 2D sequenced images from Warner Bros. 
classics such as Bugs Bunny, computer-generated 3D worlds such as Pixar's 
Toy Story, or physical stop-motion such as Aardmans Wallace & Gromit. 
Visual Effects (VFX) is the seamless use of animation techniques integrated 
within live action film, such as Jurassic Park or Lord of the Rings franchises 
and in almost every TV show on air today. Animation drives 3D and 2D 
hand-held, console, and virtual games, now immersing the player into a new 
interactive experience. 

In essence, to animate is to bring something imaginary/constructed/unreal 
to life. That something could be a wacky cartoon character, a loveable pho- 
torealistic Baby Yoda, or a perfect digital representation of a dead actor. All 
are artistic models built by creative and technical artists trained to visualize 
and interpret written descriptions, reference images, real objects, or environ- 
ments. These models are inanimate objects that when created and animated are 
imbued with the style and techniques of their creators and as such, are encul- 
turated, which is the conscious or unconscious imprinting of personal values 
and norms on digital data. As such, we must recognize the duty of care when 
bringing our vision of the past to life. 

For each platform, animation takes on a different technique, however; all 
require objects, whether 2D, digital 3D, or physical to be moved by 'keyframes; 
which represents slices of motion in time. All action within the object itself, no 
matter how large or small, must be keyframed or ‘keyed’ on a per-frame basis 
in order for that object to be animated when sequenced together. In general, 
objects are animated by frames-per-second (fps). In film, the basis of all anima- 
tion is 24fps. If the style of the animation is choppier, it is animated at 12fps. 
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For gaming, film or virtual reality (VR) that requires highly detailed or interac- 
tive movement, 48fps or higher is desirable. Why is this important? The higher 
the fps the more life-like or interactive the object appears to be. 

Motion capture is like animation, in a sense that it attempts to record the 
life-like movement of real people, animals, and objects. Like Golem in the Lord 
of the Rings feature film series, the character on film was a 3D object, animated 
by a base layer of captured movement from a real-life actor. That motion data 
is recorded in real time by hundreds of sensors and cameras within a physical 
space when the performer is acting, then combined with a 3D character object 
in virtual space. 


The Illusion of Life 
Animation 


To illuminate life is to mimic it. Several animation techniques are used to serve 
this process. They include primary, secondary, and tertiary movement, squash 
and stretch as well as over-exaggeration (see Thomas & Johnston 1995). Each 
technique is deployed at various intensities in order to trick the eye into believ- 
ing it is a real-life character or to enhance the unbelievability of cartoon char- 
acters as they distort themselves into and out of situations. 

Primary animation is the main movement of any object or character within 
2D, 3D, gaming, or virtual space. It can be a simple as a walk cycle or moving 
from one side of the screen to another. Secondary animation is the movement 
that is affected by the primary movement of the object. On a virtual human that 
could be the sway of clothing or hair. Tertiary movement is the subtle micro- 
movements of the virtual humans' muscles, eyelashes or the digital hairs on 
the skin. In combination, it is a symphony of animated gestures that help to 
convince the eye that what you are seeing is believable. 

Squash, stretch, and over-exaggerate are classical 2D cartoon animation 
techniques. If a character lands on a hard surface, they compress or squash into 
a pancake, then immediately stretch out into a long object, giving the impres- 
sion that the character is invulnerable to any danger, but when stretched it 
creates tension, which preludes a burst of speed. In over-exaggeration, a char- 
acter's gestures and movements are visually enhanced beyond reality to emote 
various forms of emotions and physical abilities. 

When using motion capture, the animation is exactly as the real actor has 
portrayed it. When applied to a virtual human character, the mind rejects 
the movement as being ‘too human. As such, motion capture animators must 
also create artist-driven secondary and tertiary, squash, stretch, and over- 
exaggeration movement layered over their perfectly captured human motion 
capture data, so that the eye is once again tricked into believing what they are 
seeing is actual human movement, when in fact it isn't. 
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Figure 7: Adventure Man (2020) by Prof. Kris Howald, Sheridan College 
Computer Animation Program - Example of texturing and shading in 
Autodesk Maya. 


Modelling, Texturing/Shading, Lighting, Rendering, 
Compositing, and VFX 


Animation is only one element within the arsenal of tools required to animate 
an object. Virtual objects need to be modelled, textured, and lighted. Once the 
right combination of texture and lighting is achieved, the animated sequence is 
rendered into still images, which again are reprocessed to further enhance the 
final image or sequence of film by the compositing of additional 2D rendered 
foreground, midground or background layers, lighting, and VFX. 

In virtual reality or interactive gaming, animation, texturing, lighting, 
compositing, and VFX are rendered in real time which requires substantive 
computing power. Unlike fully rendered 2D sequenced images, which use 
the computing power to render highly complex layers into photoreal animated 
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sequences that are then viewed non-interactively, VR and interactive gaming 
cannot achieve a photoreal image in real time and their projected models, tex- 
tures, lighting, and VFX have a less photoreal quality to them. 


Modelling 


The simplest object within any 3D space is a point. A point can contain X, 
Y, and Z coordinates, texture coordinates, normals (the front-facing direction 
of an object), animation channels, lighting coordinates, and a multitude of 
other meta- and paradata. However, a 3D point cannot be rendered unless it is 
given volume or mass. 

If you have two 3D points occupying different X, Y, and Z coordinates, a 
polygonal spline can be created, which not only connects the points together 
into a new object but also acts like an actor-network (see Carter 2017a; Latour 
2005), allowing points to share information between each other and the polyg- 
onal spline. Once three or more points are added with polygonal splines (lines) 
attaching to each point that is then connected in a loop back to the original 
point, a polygon surface or face is created in a triangular, square, or rectangular 
form. The more polygon faces an object has, the more the 3D model begins to 
have surface mass within the 3D space. 

In 3D scanning, a technique now widely used in DCH, a real object is scanned 
by light shooting from a scanning camera, registering a point in 3D space as it 
hits the object being scanned (Ahmed, Carter & Ferris 2014). As such, scanned 
3D objects have millions of points captured and formed into what is called a 
point cloud. Each point will have its own information created when the point 
is captured; however, every point within the cloud is in fact, autonomous to its 
neighbour. Only when polygonal splines begin to connect the points, either by 
hand or through an automatic software toolset, can the points share informa- 
tion such as texture and lighting coordinates and form actual 3D faces within 
the virtual environment. However, it does not capture a photographic texture. 

Photogrammetry on the other hand creates a high-resolution 3D photo- 
graphic envelope that represents the surface of an object. It cannot distinguish 
between the discreet parts of the object, so in the end what is captured is a 
‘blanket’ of visual data information. Its 3D surfaces are approximated, mean- 
ing that it is guessed by the software system. Both photogrammetry and 3D 
scanning help to provide digital archives or visual representations of original 
objects, but due to the nature of how those objects or landscapes are brought 
into 3D space they themselves cannot be animated and can only be used as 
static 3D props or backdrops, unless broken apart, rebuilt as discreet pieces, 
and given the ability to be animated. 

A model can have as many points and polygons as needed to create a 3D 
representation of the object the artist is creating; however, the more points and 
polygons, the slower the real-time rendering and final 2D rending of the image 


Animating Past Worlds 45 


is, or the more power one needs from the computers CPU and graphics card 
to interactively work with the model. This is why in 3D gaming the object and 
characters are more polygonal then in photoreal 3D objects in film, as both 
the point and the polygonal count has to be lower in order to achieve real- 
time interactive rendering. 

If creating a model from scratch isn't your forte, then open-source online 
databases such as Sketchfab (https://sketchfab.com/museums) provide a venue 
for digital cultural heritage (DCH) practitioners to upload and exchange 3D 
heritage models for reuse. 


Texturing/Shading 


Texturing and shading have two different functions within 3D modelling. Tex- 
tures on 3D objects are essentially 2D layered images that drape or wrap 
around the 3D object, or part of a 3D object. Shaders are 3D mathematical 
calculations that manipulate the actual surface of the polygon when rendered, 
thus giving textures specific properties to display. 

For instance, in modelling a wooden ship, specific textures of 2D/3D images 
applied to the surface of the ship model would simulate the detailed look of 
those objects to give it the look and feel of an actual representation of rope, 
cloth, or wood. A texture and a shader are combined to give a dull or rough 
wooden surface of a ships hull, or a shiny and reflective metal surface of the 
ships cannons or the cloth-like feel of the sails. If one wants to create the illusion 
of ocean waves on the surface of a 3D landscape, the model itself would have 
an animated noise algorithm that makes the 3D surface undulate like waves. 
The shader assigned to that surface will also have animated noise algorithms 
to simulate the secondary and tertiary movement of the micro swells, ripples, 
eddies, and other water characteristics, but also the animated colour, lighting, 
and light refraction. The shader may be reflective and thus a model boat in the 
3D water would be reflected on the surface of the water as well. 

In many of the DCH representations and photogrammetry/3D scanning 
point clouds made into polygons, grey shading is used to give the object a sense 
of weight and light displacement within 3D space and to visualize the 3D object 
for quick editing or rendering. For many DCH examples, this half-finished 
look, becomes the final representation of the object, whether due to the tech- 
nical or creative limitations of the DCH specialist. As such, the digital model 
lacks the materiality, colour, and texture, which grounds it within the context 
from where it came. 


Lighting 


Just like objects in the real world, 3D objects can be illuminated by various 
forms of light. Within 3D space, sunlight, moonlight, pitch blackness, or a misty 
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haze can all be replicated by the various types of lighting. Although 3D, virtual, 
and gaming environments may have millions of lights within their scenes, three 
basic lighting techniques - fill, spot, and ambient lighting - help to generate 
the look and feel of the 3D objects within their environment. 

Like daylight, fill light is omnidirectional, meaning within 3D space the light 
emanates all around. Spotlights are a concentrated unidirectional light, similar 
to the actual sun during daylight. Both can cast shadows by objects within 
the 3D space and work together to replicate the lighting in an environment. 
Ambient light casts soft rays in an omnidirectional mode, providing depth 
to the shaders. It casts no shadows and is used primarily for creating atmos- 
pheric lighting. 


Rendering 


There are two forms of rendering: pre-rendered and real time. Pre-rendered is 
when the 3D environment is made into a single 2D frame of film, which forms 
a complete animation of the character, object, or environment when combined 
with renders from a sequence. Pre-rendered images allow for a greater range of 
scene detail and photorealism. In real time, a gaming engine renders the objects 
or characters interactively, when in viewable range by the user. Because the 3D 
objects are being rendered in real time, the quality of the rendered image is 
solely based on the CPU and graphics card capacity to render complex scenes. 
As such, most game engine-driven renders tend to have lower-quality lighting, 
model, texture, and shader complexity. However, new hardware technologies 
are allowing for a smaller and portable form factor, combined with a higher 
CPU and graphics card capacity to eventually generate photoreal real time 
images. This is the goal for VR and augmented reality (AR) - to create immer- 
sive experiences of imaginary characters, objects, and environments within real- 
life settings. 


Compositing 


Compositing is a technique used to superimpose an image overtop or inte- 
grated within the broader 2D rendered scene. By rendering out in layers, 
artists have the ability to continue to manipulate the scene after it has been 
converted from 3D to 2D to tweak lighting, add VFX or new characters. In 
some instances, the compositing stage has been used to hand paint addi- 
tional muscles on the bodies of live actors within a scene, to make them 
look more heroic. As in 3D animation, 2D composite layers can also be ani- 
mated to enhance the secondary and tertiary motion within the overall final 
rendered sequence. 
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VFX 


Visual effects, or VFX, are generally used to increase the believability of a par- 
ticular scene or environment. Explosions, fire, smoke, sparks, water, wind, 
leaves, dust, dirt, and even crowds of background characters or spaceships fight- 
ing each other are all considered VFX. VEX is generally complex and, depend- 
ing on the integration of the VFX in the scene, can be both 3D and 2D. Within 
gaming engines, VFX tend to be suggestive as opposed to being realistic, again 
because of the complex nature of the effect and the hardware rending available. 


Tools and Techniques 


To animate, one first needs to build an object, such as a character, in which to 
add motion. In 3D animation and gaming, this process starts with the model- 
ling of the object within software applications such as Side Effects Software's 
Houdini, Autodesk’s Maya, or Blender 3D animation software. Textures, shad- 
ers, and animation rigs are applied to the models and those animated models or 
static 3D objects/assets are placed within a larger 3D environment with other 
character and objects. 

If the 3D environment is going to be user-driven, as in a 3D game, then the 
prebuilt 3D objects/assets are typically imported into either a Unity or Unreal 
game engine. At this point, the 3D objects/assets, if given the ability to be 
manipulated by the user using a game controller or haptic device, can then 
be moved, picked up, or affected interactively. If the 3D environment is part 
of an animated sequence and is intended as a passive 2D viewing experience, 
the 3D animation is rendered out to a sequence of 2D images and then into 
a video. If the intent is for the user to be completely immersed within the 3D 
environment, again a game engine is used to drive the user interactivity con- 
trols; however, the image is projected into a VR or AR headset as opposed to 
being rendered in real-time to a 2D screen. As such the user’s physical motion 
is tracked and replicated within the VR/AR environment, allowing for the illu- 
sion of full immersion. 

In some cases, handheld devices such as phones and tablets can be used to 
visualize 3D objects within physical spaces, but again the ability to see photore- 
alistic images are limited by the CPU and graphics card capabilities. 


Technical and Artistic Limitations 


For most in the DCH field, the immediate technical limitations start with the 
laptop or desktop being used. To render and display 3D objects in real time, 
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the computer needs a robust level of RAM, as well as a fast graphics card and 
CPU. Further, most animation software applications function best in a PC or 
Linux operating environment. In many cases, DCH funding does not allow for 
large-scale equipment purchases and DCH specialists. Multifunctional laptops, 
which tend not to be animation production-ready, are more practical in day-to- 
day use but not necessarily suitable for 3D animation creation. 

Although almost all 3D animation software packages now offer a free student 
version for personal/research work, many of these applications also reduce 
the functionality, forcing students and researchers to eventually buy, lease, or 
borrow full-functioning tools to complete complex tasks or desired VFX not 
allowed in the free versions. Further, very few DCH academic programs teach 
3D production other than a cursory introduction or not at all. As such, DCH 
hopefuls must hack their way through the steep learning curve to achieve basic 
3D animation skills required to do their research or 3D projects. 

Private DCH production companies such as LithodomosVR (www.litho 
domosvr.com) have deployed traditional film, TV, and animation production 
techniques with consumer-level technology to create vibrant, materialized (re) 
visualizations of ancient worlds. These teams combine creative, technical, and 
subject matter experts, who work in traditional animation production pipe- 
lines, within the constraints of the software and the delivery platforms. VR and 
AR technologies at the consumer or prosumer level are at an effective price- 
point for DCH activities; however, budding developers quickly discover that 
functionality and features are inadequate in both ease of use and for the com- 
plexities of DCH entry-user needs. 


Example 


The DCH-making process is a wayfaring path (Ingold 2011). The tools, the 
material, the available data and knowledge, the moment in time, and ones skill 
all contribute to the construction of new knowledge and with that, a (re)vision- 
ing of the ancient past (Carter 2017c). Recent examples of 3D DCH produc- 
tion and research demonstrate a shift towards a professional theoretically and 
creatively grounded practice. 

The Slingsby Castle reconstruction by University of York Master's graduate 
Bethany Watrous (Watrous 2018), with its highly curated and researched (re) 
visualization, demonstrates a new wave of academically trained 3D artist/DCH 
specialists who are rising to the challenge of bringing the past into the future. 
Watrous and a larger cohort of recent graduates in DCH represent a new form 
of 3D DCH production, where guidelines such as The London Charter and the 
use of paradata give promise to a theoretically grounded but animation-centric 
visualization of the past (see Denard 2012). 

Professional 3D animation artists such as Bob Marshall represent a group of 
DCH creative practitioners who were traditionally trained in animation and 
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Figure 8: Warwolf - The Siege of Stirling Castle, 1304 by Bob Marshall. 


then acquired their DCH knowledge through years of experience and pro- 
ject commissions. In Warwolf - The Siege of Stirling Castle, 1304 (see Figure 8, 
above), he uses complex lighting, VFX, and character animation, combined 
with historical and archaeological data, to illustrate the tension and atmosphere 
in a slice of known history. 

Beyond just visualizing within virtual space, Jonathan Westin (www.melting 
history.org) from the Center for Digital Humanities at Gothenburg University, 
recently travelled in January 2020 with a team to the Antarctic to 3D scan, 
using a FARO system, a historic Swedish research outpost before it is lost to 
global warming. Apart from capturing the archaeological site digitally, the team 
also recorded light and specifically sound, to further incorporate within their 
future 3D virtual reconstruction. This process, which they called phenome- 
nology, immerses the viewer deeper into the virtual world by adding, sound, 
touch, and smell (Cooper 2019; Jeffrey 2015). 

Ubisofts Assassins Creed video game franchise, although not strictly DCH, 
represents an achievement in the application of 3D animation within a 
DCH-inspired virtual environments (see Ávila, Corso & Fischer 2020). The 
gaming company has taken a proactive community-engaged approach when 
conducting background research on any of their titles that deal with cultures 
not their own. Ubisoft actively employs a historian and a team dually trained 
DCH specialists to support the attempt of historical and archaeological accu- 
racy within the franchise. They also have the budget and resources to digitally 
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capture existing historical architecture, which is then used as 3D assets within 
the games. As a result of the recent loss of the structural integrity of Notre Dame 
due to fire in 2019, Ubisoft's 3D-scanned interiors and exteriors have played a 
part in the restoration planning and process to slowly rebuild the cathedral to 
cultural heritage standards (Avila, Corso & Fischer 2020). 


Discussion 


The birth of virtual/digital archaeology can trace its roots back to 1987, when 
mainframe computers were required to render even the simplest grey-shaded 
object to screen (see Reilly & Richards 1987; Reilly 1991). Desktop computers 
would take almost another decade to become ubiquitous, and it would take 
even longer for the cultural heritage community to embrace this new technol- 
ogy. Many DCH examples over the last four decades are still missing the engag- 
ing and life-giving qualities of animation. We now compete with the public and 
institutional desire to consume high-quality digital media. Whether due to the 
lack of artistic or technical animation skills, access to technology, or the cost 
to produce, DCH continues to be overshadowed by the robust technical and 
creative advances and substantive budgets in the entertainment industry and as 
such, our own inability to fulfil those expectations. 

Unlike entertainment-based animation, where the digital assets, environ- 
ments, and digital people are the creation of the writers, producers, and artists 
who are free to mix and mingle historical or fantasy influences, DCH practi- 
tioners have a ‘duty of care’ in representing the cultural, historical, or archae- 
ological unknown (see Perry & Taylor 2018; Huggett, 2015, 2017, 2018)held 
in Oslo. The theme of CAA2016 was ‘Exploring Oceans of Data, alluding to 
one of the greatest challenges in this field: the use and reuse of large datasets 
that result both from digitalisation and digital documentation of excavations 
and surveys. 

What does this mean? Any historical representations of past cultures, 
whether Roman, Mayan, or Nubian, are based on academic speculation, 
surviving artwork that may depict how certain objects or people looked, or 
colonial accounts, which are for the most part are laced with highly racist 
and stereotypical representations of the ‘other. As such, DCH practitioners in 
many cases (re)visualize the historical unknown. 

When there is a gap in the knowledge, an educated guess is made and 
then visualized. Unlike entertainment-based artists, DCH practitioners must 
explain how they filled the gap in knowledge, what sources they used, and if 
no sources were available, how they were able to construct new knowledge and 
present an alternative (re)visualization. This is called 'paradata which in effect 
is a recoded diary of all of the decision-making choices a DCH practitioner 
makes to build or construct new 3D objects, data or knowledge (Baker 2012; 
Bentkowska-Kafel, Denard & Baker 2012; Denard 2012). Thus, our duty of care 
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is to not only visualize the cultural, historical, or archaeological unknown, but 
to also provide an explanation as to how one arrived at making that digital 
representation, in the most ethical, equitable, culturally considerate, and scien- 
tifically enabled way (see Dennis 2020) 


Considerations 


On the horizon, digital cultural heritage promises to be a scientifically and cul- 
turally valuable asset in representing the cultural, historical, or archaeological 
unknown. However, the potential ability to create 3D models in software pack- 
ages such as Blender, or to access and reuse open access heritage 3D assets in 
Sketchfab, or the ability to combine assets within a real-time game engine such 
as Unity or Unreal then interactively engage with them in virtual space through 
an Oculus or HTC VR system, hasnt become easier for either novice profes- 
sionals or consumers yet. Creative and technical skills acquired through being 
self-taught or through additional technical animation courses, combined with 
cultural heritage training, is still required to bring past worlds to life. 

Over the last 10 years, universities such as the University of Glasgow, Uni- 
versity of York, University of Dundee, Bournemouth University and especially 
Glasgow School of Art have embraced digital cultural heritage and now have 
actively combined programs and courses in animation and cultural heritage. 
As such, digital technology practice and the heritage sector have begun to con- 
verge, and this has created a new generation of digital cultural practitioners. 

One of the greatest risks digital cultural heritage specialists face today, is the 
pervasive hyper-realism of the constructed 3D image. As software, hardware, 
and actual skill sets have been simplified, both professionals and consumers 
can, with some practice, create life-like images of ancient landscapes, peo- 
ples, artefacts, UFOs, and dinosaurs. The mundane can now be digitally rep- 
licated in uncanny realism (see Mori, MacDorman & Kageki 2012). As such, 
those constructed 3D environments can be misconstrued as being 'real; and 
there lies the danger of being able to visualize or reconstruct the ‘unknown’ 
without providing the context or research to support the (re)visualization 
being presented. 

Conversely, should DCH specialists be the sole authority on what should and 
shouldn't be digitally culturally represented? What about indigenous, people 
of colour, or colonized voices? Does this technology allow underrepresented 
community stakeholders to also visualize their perceived pasts, in a manner 
representative of their cultural considerations? Who decides what is culturally, 
historically, or archaeologically accurate? 

Large game companies, such as Ubisoft, now actively create new cultural- 
historical entertainment products that not only (re)visualize the ancient past, 
but retell stories and narratives in a convincing and engaging way. Ubisoft, in 
particular, has gone to great lengths to include community engagement when 
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representing ancient peoples, cultures, objects, and landscapes in their histori- 
cally influenced Assassins Creed gaming franchise. But what happens in the 
future when this approach is too expensive or accidentally or intentionally 
misrepresents history in a way we know is categorically untrue? What if the 
consumers of these entertainment games, TV series and films begin to believe 
in what they are seeing is a real representation of history, only due to the fact 
that the digital (re)visualization of people and places is so effectively lifelike? 

Lastly, what is our role in digital cultural heritage? How, as potential and 
active practitioners, can we engage reflexively, thoughtfully, creatively, techni- 
cally, and academically within this ever-expanding field through the creative 
and communicative medium of animation? What is our ‘duty of care and how 
do we ensure community involvement and engagement when representing past 
histories from a culture that is not of our own? 
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Abstract 


How does ‘digital’ apply to ancient pasts? Digital methods, especially meth- 
ods relating to identifying, visualizing, and analysing spatial data, have become 
increasingly important within the fields of classical literature, archaeology, and 
heritage. On the one hand, literary narratives offer potentially different ways 
of representing space and place than the usual cartographic maps to which we 
have become accustomed. On the other hand, by virtue of being able to locate 
cultural artefacts in space - where they were found, through whose hands they 
have passed, where they reside now, where they were produced and circulated 
- it becomes possible to construct biographies or even itineraries of objects that 
offer richer ways of understanding their use and agency. 

Unique in all classical literature, Pausaniass second-century CE Periegesis 
Hellados presents an example of both types of spatial representation - a nar- 
rative that describes places of interest in the Greek landscape as well as the 
notable objects found there. This chapter discusses some of the ways in which 
Pausanias narrative of Greek heritage is good to consider when attempting 
to use digital methods for analysing the entanglements of place, people, and 
objects in a textual geography. 


Introduction 


Digital cartography and geographic information systems (henceforth GIS) 
have been used for decades to connect primary sources (such as literary, 
archaeological, historical, and heritage evidence) to spatial data and further 
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visualise them in cartographic interfaces. The spatial turn in humanities disci- 
plines has been extensively applied to the study of ancient lifeworlds and has 
become increasingly important within the fields of classical philology, archae- 
ology, and heritage (Barker et al. 2010; 2012: 185-200; Lundqvist & Landeschi 
2015: 25-27). 

GIS is a technological framework for gathering, storing, managing, presenting, 
and analysing data systematically, as a scientific method (Dunn 2019; Foka et al. 
2020: 203-217). Exploring the spatial narratives of objects and peoples opens the 
possibility to a deeper and broader understanding of the past, where and when 
they were situated in history as space becomes place, imbued with meaning. 

The Barrington Atlas became a modern GIS in 2000, covering Ancient Medi- 
terranean geography, but literary territories are limited to the study of topo- 
nyms (place names derived from topographical features). Paladino (2016) notes 
how the semantic concept of space is not necessarily limited to routes; it can be 
seas, islands, or lakes. Other words too, may have semantic importance beyond 
their geographical locations. The importance of engaging with the geographies 
of artefacts as they transcend through histories and cultures to provide readers 
with a fuller analysis of provenance ought to be highlighted as object itineraries 
(Dunn et al. 2019: 253-271). GIS, however, may be limited to the annotation of 
place as static toponyms, as they do not classify other concepts or entities, such 
as temporal data, social networks, or movement for example. 

Here we focus on the complexity of mining spatial heritage datasets by look- 
ing at mining information from languages beyond English. Our case study is 
Pausanias’s Periegesis Hellados (Description of Greece),' a cultural geography 
of Greece written in the second century CE in Greek and composed essentially 
by 'the things that deserve to be recorded' (Pausanias, Description of Greece 
1.39.3). In this we ought to note that Pausanias wrote at a time of Roman colo- 
nisation of Hellenic space and was, as it transpires from his work, particularly 
interested, even biased, to illuminate his readers on Hellenism and its history 
across the Mediterranean. 

Periegesis has been widely used as a guide within the field of classical archae- 
ology, relating to archaeological or monument locations but further also 
connecting to the movement and population of sites as well as artefact trans- 
portation and composition. It has been, for centuries, treated as an information 
repository, particularly for the discovery and interpretation of peoples, sites, 
and, subsequently, for Hellenic heritage artefacts and monuments. The com- 
plexity of his description as well as his selective working methods have led to 
several misunderstandings. 

At the dawn of the 20th century, Willamowitz’s peripatetic archaeological 
methods had rejected Pausanias’s topography as inaccurate and biased at best. 
While there are certainly locations described in a selective and biased man- 
ner, Pausanias topographical descriptions of larger sites more often match the 
archaeological record, as demonstrated by the sanctuary of Apollo at Delphi by 
the Ecole Française d'Athénes and the Athenian Agora by the American School 


58 Virtual Heritage 


of Classical Studies (Cundy 2018: 3). However, Pausanias's description of place 
does not necessarily map easily on to the archaeological record that emerges 
through excavation (see Hutton 2005). Instead, Pausanias topographical narra- 
tive entails past accounts of the places through which he passes and the objects 
in space he sees. It is a narrative time machine of sorts, a highly selective pro- 
cess that binds together place, artefact, origin, and purpose in space in dispa- 
rate historical instances the author narrates. 

Epistemologically, the complexity of this time-space matrix illuminates the 
time-depth problem of the Greek East - that is, by providing ways of com- 
partmentalizing and marking these "different temporalities of the long-study 
of urbanism in the Mediterranean East' (Hodder 1993: 279-80; Stewart 2013: 
236). Pausaniass Periegesis is a manifestation of literary territories as con- 
ceptual and subjective, comprising a specific selection of data. The historic- 
geographic method has been criticized for the loss of subjective and local varia- 
tion (Cocq 2008). Similar concerns have been raised by some archaeologists on 
GIS usage in archaeology, including the suggestion that the technology removes 
the human, subjective aspects of interpreting data (Conolly & Lake 2006; see 
Vassalo et al. 2018 for a review of discussions with respect to the use of 3D GIS). 

Nature and culture in their tangible (settlements, artefacts, people) and 
intangible (movement) forms are thus studied here as spatial extensions 
and networks of disparate data. The application of data science and information 
systems does not remove the complexities associated with traditional cartogra- 
phy and even introduces new challenges. The aforementioned case study thus 
helps identify the validity of digital methods to understand the spatial dimen- 
sions of ancient narratives as a research, educational, and dissemination tool. In 
foregrounding the role of digital technology, this research takes, as its starting 
point, the inherent statistical bias. 


Pausanias' Role: Extracting Heritage Data 
with Computational Methods 


Heritage more generally, has come to mean the events, materials, or processes 
that have a special meaning for the memory and identity of certain groups of 
people. Heritage is a concept that springs from modernity's ambitions in order- 
ing, classifying, and categorising, but also the idea ofa threat or a risk that forces 
humanity to recognise identities and their tangible or intangible representation 
(Harrison 2013). Previous classical scholarship (including archaeology and 
reception) has examined the text in terms of: narratology of heritage (Akujarvi 
2005; Habicht 1985; Pretzler 2007); identity and memory (Alcock 2001; Arafat 
1996); and ethnicity and religion (Konstan 2001). Pausanias' spatial description 
of the towns, buildings, and monuments through which the reader moves has 
been widely used as a guide for interpreting those sites and their archaeology 
and classical heritage (Dyson 1988; 2006: 79, 251-254; Shanks 1996: 49-52. 
Cf. Habicht 1985: 70-77. Cf. Stuart 2013: 236). 


Mapping Ancient Heritage Narratives with Digital Tools 59 


We discuss here the potential for a digital conversion and rendering of 
this spatial narrative of Greek monuments and artefacts, using a number 
of methods, such as Recogito, a platform for semantic annotation of text as 
well as exploring the possibility for text mining, to tease apart the relationship 
between movement, space, and memory. In doing so, we group our inquiries 
into themes, notably examining Pausaniass description of locations of mem- 
ory through geovisualisation, looking further at the potential for extracting 
dynamic relations of movement or origin. In what follows, we discuss previous 
scholarly attempts to geovisualise ancient narratives with digital technology, 
the complexity of mining Periegesis Hellados for spatial data, our close reading 
data-gathering methods, and our semantic annotation strategy and tools, nota- 
bly the platform Recogito, and future plans. 


Mapping Meaningful Journeys in Contemporary 
Cartographic Environments 


Geographic in this case means a georeference, an actual location on the Earth, 
a place that can be represented on a cartographic map. There are a number of 
complexities associated with this endeavour. First, a location described within 
a text may have a mythological location. One example is ‘Hyperborea, which 
is a mythical ‘northern’ (assuming of Greece) location that Pausanias refers to 
multiple times within the text and in relation to other real places. According 
to Pausaniass Description of Greece, 5. 7. 8, Hyperboreans were people who 
lived above Boreas, another name for Thrace, but in maps based on points and 
descriptions given by Strabo Hyperborea, shown interchangeably as a penin- 
sula or an island (Strabo 11.4.3). This makes the place altogether impossible 
to locate as a point on a conventional map. Within the Recogito built-in maps, 
Hyperborea is only conventionally located (Figure 10). 

A second issue is the very temporality of cartographic environments. A 
space becomes a place because of specific temporal parameters. Whole towns 
relocate and change names over time, and often colonies have identical names 
to the ‘motherland. One example of this in Pausanias is the town of Achaia, 
which according to Pausanias (7.1.1) himself refers to the land between Elis 
and Sicyonia, reaching down to the eastern sea, in his contemporaneity called 
Achaia after the inhabitants, but previously named Aegialus. Another meth- 
odological issue is using temporal data; libraries for the parsing of ancient dates 
are scarce and incomplete, so the present options are to draw upon time period 
and data gazetteers such as PeriodO (http://www.perio.do) and Trismegistos 
(https://www.trismegistos.org). 

A third issue is environmental change. For example, in a contemporary map 
one might not be able to identify a now-submerged island or a drought river. One 
example is rivers in Asia minor that, while discussed in Pausanias’s book seven, 
‘cannot be identified in the digital atlas of the Roman empire, or even the sub- 
merged island Vordonisi in the sea of Marmara. Thus, mapping an already rich 
text with heritage data into a cartographic environment becomes challenging. The 
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Figure 9: Greek and English translation of passage 1.18.5 of Pausanias's 
Periegesis Hellados with different relational annotations to display movement 
and origin in the semantic annotation platform Recogito. This displays the 
complexity of mapping narratives of heritage in different languages. 
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Figure 10: A conventional (and incorrect) mapping location for mythical 
Hyperborea as found in the built-in cartographic system of Recogito. 


scholar must decide how this may be represented in a spatial manner. The deci- 
sion may even be made to not represent it. However, a decision must be taken. 


Working Method: Recogito and Some Preliminary Results 


Currently, we have uploaded Pausanias’s ten books to the local Umea univer- 
sity servers instance of Recogito (http://recogito.humlab.umu.se) in order to 
curate the document as a database of heritage-spatial information. The working 
method is to align words to appropriate georeferenced data, found in several 
gazetteers. The most complete gazetteer for spatial data of the Balkan penin- 
sula and the Eastern Mediterranean is Pleiades. For more granular topographic 
and heritage data we additionally use Topostext gazetteer. For art historical 
artefacts and monuments finds we use Judith Binders Art History Gazetteer 
and the German Archaeological Institutes (DAI) gazetteer for archaeologi- 
cal finds (e.g., districts, temples, statues, etc.). If no appropriate match can be 
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Figure 11: Recogito interface with marked- up, identified, and DARE gazetteer 
alignment of the Ancient Temple of Hera in the island of Samos (Heraion) 
including additional free tags such as ‘build’ and ‘temple’ 


found, then we use the yellow flag option and the comments box for further 
details that are then returned to the gazetteer developers. Recogito supports 
further free tagging, that is, enriching each word with additional information 
(see Figure 11). 

To this, we have a working ontology: a structured list of information pertain- 
ing to structuring and organising cultural heritage words and then enriching 
them with metadata. For marking spatial entities, we use the Place entity tab 
to mark the place in the document. Then, when the pop-up box appears, we 
align the place reference to an appropriate gazetteer entry using the map or 
Recogito’s automatic matching. 


» If the place represents a human footprint on the landscape that denotes her- 
itage data (e.g., city, temple, etc.), we use ‘built’ to enrich the word. 

» If the place represents a physical feature of the landscape (river, sea, moun- 
tain, etc.), we use ‘physical. 

e If the place represents a conceptual area or territory (e.g., Messenia, ‘the 
Corinthian land’ (chora), ‘Greece’ etc.), we use ‘regional. 

e If the place represents a clearly mythical space (e.g, Hades), we use 
‘mythical. 

e if the place represents an object in space (e.g., statue, xoanon [wooden 
image], dedication, column, etc.), we use object: 

e If the place represents a material (e.g., Phrygian marble, Assyrian fabric) 
from a provenance other than the object or building it belongs to, we use 
‘material’ - and use relational tagging > ‘provenance. 


We further use a second tag: 


e For built, further defined as: "settlement, ‘temple; etc.; for physical, further 
defined as: ‘river’ ‘sea, ‘mountain, etc. 
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Figure 12: Object identification and tagging in book 7 section 4 of Periegesis, 
including mark-up and georeference alignment in the DAI gazetteer. 


e For regional, further define if further information is given (e.g., ‘deme’ ‘the 
Corinthian land' (chóra) etc.). 

e For mythical, further define using the place mentioned: e.g., ‘Hades. 

e When it comes to heritage objects we further define as: ‘agalma, *xoanon; 
'anathéma, ‘kion, etc. (see Figure 12). 


Producing a geo-annotated version of Pausanias’s Description of Greece means 
that we will identify and analyse the forms of space within Pausaniass narra- 
tive - the ways in which places, monuments, and other objects (heritage data 
such as sculptures, tombs, etc.) are described in the text, and how the narrative 
is organized spatially. Using Recogito, the team semantically annotates ‘places’ 
using the following method. First, an entity is identified in the text as a place (or 
object in space). Then it is mapped (where possible) to a global gazetteer. Third, 
tags are used to provide additional information for, and construct a schema 
for thinking about, place in more depth, such as: whether the place is physical 
(a river, mountain, etc.), built (a city, temple, altar, etc.), regional (a wider geo- 
graphical area), or mythical. 

Recogito further supports a range of export formats that can be enriched 
with additional information as metadata. 'The options presented in our private 
instance relate to downloading annotations in CSV, as a data table for importing 
into spreadsheet software or a GIS. There is a further possibility to download 
annotations and document metadata as RDF, encoded using Open Annotation 
and Dublin Core, in JSON-LD, RDF/Turtle and RDF/XML formats. For places, 
the user is able to download confirmed geo-located places in the document as a 
GeoJSON FeatureCollection. Geo-located places can finally be downloaded as 
a KML file, for viewing in Google Earth, for example. We take advantage of this 
aspect of Recogito to not limit data reuse from the project based upon the GIS 
or software system one uses. 
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Text Mining Possibilities 


When mining text there are considerations that must be taken into account; 
decisions about these considerations must be made and then acted upon. The 
first consideration is the original language of the text. Will the text mining take 
place across the original language or will it occur across the language to which 
the text has been translated into? In this situation, the original language of the 
text is Greek. A decision must be made regarding mining the Greek or, for 
example, the English translation. If it is decided to mine the English translation, 
this must be documented to clearly identify to the reader that it was a transla- 
tion and not the original that was mined. 

If the original language is to be mined, then this will influence the applied 
text mining method and algorithm selection. Although this may be obvious to 
the skilled linguist, it may not be obvious to a novelist and may lead to incorrect 
conclusions about what text does and does not contain. Different languages are 
constructed in different ways and hence, in text mining, these must be consid- 
ered and incorporated into the algorithms to be used. 

Here Greek is the original text's language, while English is the book chapter 
language. Greek is constructed differently from English and hence the text 
mining algorithms must take this into consideration. Again, perhaps obvious 
to the linguist but perhaps not obvious to the spatial or computer scientist. In 
the case of Greek, an inflected language that utilises cases that alter the suffix 
of the given noun instead of prepositions, the question of text mining becomes 
a complicated issue. For example, if one needs to mine the term for sculpture, 
that is ‘agalma’ (&yaAuo), finding noun in another format (e.g., in genitive pos- 
sessive: åyáuatoç) means that one needs to mine another version of the word 
that is significantly changed - perhaps for Greek only focusing on the stem, 
excluding the suffix that denotes a case, for example, &yóAp. 

The construction of a system permitting automated mining and comparison 
of the original and translated texts may further enhance the understanding of 
what the text contains and may highlight possible current era misunderstand- 
ings of a text's content. 


Conclusions: Dynamic Relations, Spatial Complexity and the Future 


Within the project Digital Periegesis, the task is to enrich character strings, 
words that have a semantic purpose with spatial data. In this chapter, we dis- 
cussed the possibilities and complexity for discovering words that denote herit- 
age and then enriching them with relevant data. The team tested several ways to 
mine and ascribe metadata, most notably working with the semantic platform 
Recogito. In spite of the complexities in close reading of the text, it is possible, 
using appropriate cartographic environments and gazetteers, to create a digi- 
tally enriched edition of Pausanias’s description of Greece. The combination 
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of a number of gazetteers for the ancient world as well as the use of an accu- 
rate cartographic environment makes the exercise of semantic annotation 
in Recogito a deep learning process of Hellenic heritage across and beyond 
the Mediterranean. 

During the process of annotation, the importance of data relating to time 
and people was noticed. It was not only the case that people were being used 
as proxies for places; Pausanias also showed interest in inventories of people, 
either by ethnicity, by historical or mythical means, or even by genealogy, as 
an alternative means of structuring his information. Most importantly, it was 
observed that Pausanias not only moved through space but through time. LOD 
methods and tools in the Digital Humanities, however, are currently limited 
to place. The lack of appropriate LOD ‘authority files’ for temporal and pros- 
opographical data entities can be further investigated in the future, as well 
as the possibilities for text mining spatial heritage information. Using the 
techniques described within, it may be possible to create an interface to map 
spatial information and consequently, a symbology that will fit the purpose 
of creating visual maps for historical geovisualisation for Hellenic heritage 
more generally. 


Notes 


! The project Digital Periegesis (www.periegesis.org) is a collaboration 
between Humlab at Umeå University, Uppsala University, The Austrian 
Institute of Technology, The Open University in the UK and the Pelagios 
Network of Partners. It is funded by the M&A Wallenberg foundation 
(2018-22). 
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Abstract 


The importance of physical and tangible qualities in museum visits has been 
established by extensive literature exploring the importance of materiality 
(Dudley 2013) and multisensory experiences (Levent & Pascual-Leone 2014) 
of heritage. A challenge for digital technology design is to ensure that these 
dimensions are not lost to visually heavy virtual experiences. This chapter 
examines hybrid interactions in museums, outlining exemplars of success- 
ful physical-digital installations and defining the key aspects to consider for 
their design and evaluation. The goal is to complement chapters on virtual 
approaches to heritage with insights on how and why to successfully bridge the 
physical and the digital in hybrid designs. 


Introduction 


Museums are still for the most part physical places, where heritage objects and 
environments are displayed, and where even intangible heritage is exhibited 
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and made available also as part of physical exhibitions (Bortolotto 2007). 
Despite this, interpretation strategies very often neglect the materiality that 
characterises the experience of heritage. 

The importance of materiality in museums has long been stressed by some 
heritage scholars and practitioners; this includes the materiality of spaces, of 
artefacts, but also the sense of bodily immersion and close contact with the past 
that add to a memorable visitor experience (Falk 2009). These aspects are often 
elusive, and visually and textually heavy interpretation tools (i.e., information 
panels, labels, illustrative videos, and other similar devices) have trouble cap- 
turing and conveying them. 

Sandra Dudley (2013) made the argument for re-thinking the nature and 
general approach to education and interpretation that museums have had 
and that gives primacy to visual and cultural content, in favour of an approach 
in support of materiality. She defines materiality as the 'summation of physical 
characteristics, sensory experience and meaning’ (Dudley 2013: 15) of heritage, 
and therefore as a human-centred concept capturing not only physical heritage 
assets, but the way in which they are experienced, understood, and felt. This 
is extremely relevant to the use of digital and virtual heritage applications in 
museums, as technology design has long followed a similar approach to that 
applied to ‘traditional’ interpretation: ‘the “information over object” approach 
has influenced also the use of digital technology in cultural heritage ever since 
computers started to populate the exhibit floor’ (Petrelli et al. 2013: 60). 

This concern about the limited focus on the materiality of heritage assets is 
also at the core of critiques of some approaches to virtual heritage that con- 
sider virtual platforms as ultimate solutions to suit digital interpretation needs 
(Petrelli et al. 2013). Technologies such as VR have shown their worth in many 
instances, particularly in bringing engaging games and educational narratives to 
life (Champion 2011, 2015; Roussou 2004), but some problematic issues emerged 
when they were evaluated (Gillam 2017; Pujol-Tost & Economou 2007). 

An approach to extensive virtualisation can indeed be invaluable, for example 
when sites or objects are no longer existing or accessible; however, it can cause 
drawbacks, particularly in certain heritage contexts such as historic buildings, 
or living history sites, where engagement is not just about specific artefacts, but 
about the atmosphere of a place, including the sounds and smells surrounding 
visitors, which the use of digital means of interpretation should also be sensi- 
tive to. While virtual heritage applications can enable experiences that would 
not be possible otherwise (such as the exploration of reconstructed sites and 
objects that are lost or not easily accessed), the risk is that some of the material 
aspects of heritage are too readily erased or excluded from the design process. 
This is particularly critical for those heritage sites and those museums that rely 
particularly on tangibility, such as, for example, living history sites. 

Augmented reality (AR) has been an important step forward to recontex- 
tualising the virtual in relation to objects and to the environment (Beheshti 
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et al. 2017), providing a way to embed virtual content in embodied and multi- 
sensory visitor experiences (Keil et al. 2013). 

However, there is room for more experimentation with hybrid approaches 
(Bannon et al. 2005), where the virtual and the material (and the design 
thereof) are more tightly entwined. Hybrid approaches not only can be more 
mindful of augmenting (rather than substituting) material and tangible char- 
acteristics but can also extend the potential of digital technologies to a wider 
range of visitors and visitor experiences. For example, Dudley illustrates how 
powerful ‘physical, real-time, sensory engagements - even those which may 
be imagined - with material things’ (ibid.: 5) can be, and she argues that the 
material per se can be engaging where there are cultural barriers to, or limited 
interest in, other avenues of interpretation: ‘through our sensory experience of 
them, objects have some potential for value and significance in their own right 
(Dudley 2013: 8). 

Museum visits are of course multi-sensory. Notably, the importance of touch 
has been widely recognised in heritage interpretation research and practice 
(Classen 2005), and it is something that can only be partially replicated through 
haptic technologies. There have been also examples where certain smells and 
foods to be tasted have been used to accompany exhibits (Levent & Pascual- 
Leone 2014). A notable example is the Sensorium exhibition at Tate Britain 
in London (Davis 2015), where several paintings in the museum were paired 
with multisensory exhibition content to be experienced together and comple- 
mentarily by visitors. Figure in a Landscape by Francis Bacon was accompanied 
by an immersive soundscape and an olfactory display of bitter chocolate. Full 
Stop by John Latham was complemented by the sound of a heavy downpour 
and an ultrasound haptic device that made visitors feel the sensation of falling 
raindrops onto their hands (Vi et al. 2017). 

Furthermore, it is important to consider tangible, material aspects of visiting 
museums not only in relation to specific artefacts (e.g., a painting, sculpture, his- 
toric space, etc.), but in terms of how they shape the experience of the wider con- 
text (e.g., a sequence of exhibits and the interrelations among them), and of the 
presence of others (e.g., companions, co-visitors, or other people who happen to 
be in the same space). Physical co-location is still key to engender and support 
social interaction: not only in terms of people being able to talk to or be close to 
companions, but also to be aware of others' physical proximity and presence in 
planning and practicing ones next moves, and spacing, pacing, and peripheral 
interaction in the exhibition space (Heath et al. 2002; Hornecker 2010, 2016). 

The importance of materiality in digital heritage experiences extends to yet 
another aspect: the devices or artefacts that convey virtual content have their 
role to play as physical artefacts. The form factor, material, and feel of digital or 
hybrid devices also shape the experience of virtual content. For example, stud- 
ies of tabletop installations (Block et al. 2015; Hornecker 2008) have shown that 
peoples ability to cluster and gather around the display surface, and the gestures 
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that they learn and develop to interact with it, are as important in delivering 
a positive experience as the virtual content that is presented and the way that 
it is displayed (Hinrichs & Carpendale 2011). A comparative evaluation study 
of a mobile phone app, smart cards, and augmented replica objects created to 
convey the same content in an interactive exhibition showed that the tangible 
means of interacting (cards and replica objects) were favoured by visitors of all 
ages when compared to using the mobile app (Petrelli & O'Brien 2018). 

Overall, there is ample scope and definite potential to experiment with more 
hybrid virtual-physical forms where successful immersivity (Kidd 2017) is not 
obtained by surrounding visitors with virtual content, but by engaging narra- 
tives that blend the digital and the material. For all these reasons, there have 
been many explorations of interactions bridging physical and virtual: notable 
examples are mixed reality (Benford & Giannachi 2011), tangible interaction 
(Hornecker & Buur 2006), and hybrid design where physical and virtual com- 
ponents are crafted together (Bannon et al. 2005). 

Furthermore, there have been different approaches to designing actual inter- 
activity in the context of these hybrid experiences: while with virtual reality, 
quite often the metaphor is that of traveling (in space or time) or of stepping into 
a different reality. Interactivity with hybrid installations is more ‘digging deeper’ 
in the here and now, revealing qualities or aspects of spaces or objects that 
are being held, handled, or occupied in real time. These forms of hybrid inter- 
activity can be articulated in various ways. 


Approaches to Designing Hybrid Interactivity 


Broadly, hybridity means that the interactivity blends the virtual with the phy- 
sical and material; however, this can take different forms and therefore lead 
to different approaches to design affecting different sides of the experience 
of heritage. We identify four broad approaches to designing hybrid interac- 
tivity: virtual-physical overlay; hybrid objects; virtual-physical assembly; and 
hybrid takeaways. 


Virtual-Physical Overlay 


In a virtual-physical overlay design approach, the virtual and material layers 
overlap in some way in real time. A well-known example of this is augmented 
reality (AR), where the visitor unearths the virtual content by means of a ‘see- 
through’ device. This can be an off-the-shelf device such as a mobile phone or 
tablet, but also a specially crafted object with embedded electronics, whose 
physical form supports interactivity in different ways. One example is The Loupe 
(van Dijk 2019; van der Vaart & Damala 2015), where the AR device took 
the form of a magnifying glass, and the virtual content was triggered not just 
by pointing The Loupe towards an object, but also by handling it through a 
range of physical gestures that made it behave in different ways. 
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Another example of virtual over physical overlay are projection mapping 
installations. These can be realised on walls, or in entire rooms, such as for 
example in the Immersion Room installation at the Cooper Hewitt Smithso- 
nian Design Museum in New York City (USA) (https://www.cooperhewitt.org 
/events/current-exhibitions/immersion-room/). Bespoke physical structures 
can be designed to be not only a projection surface for the virtual content, but 
to carry meaning and to shape the way visitors physically arrange themselves in 
relation to the installation and how they interact with it. A well-known example 
of this that was mentioned earlier are tabletop interactives (Hornecker 2008), 
which allow for multi-user interaction, shifts in physical orientation, and coop- 
erative behaviours around the table (Hinrichs & Carpendale 2011). Another 
example in commercial exhibition design is the Weaving Time installation on 
traditional weaving patterns, realised as part of an exhibition on the Inca civili- 
sation held at Pointe-à-Calliére Museum in Montreal (Canada). In this case, the 
virtual content is displayed on and can be interacted with through a physical 
structure replicating a loom (https://gagarin.is/news/designing-tangible-inter- 
actions-from-across-the-ocean). In these examples of overlay, the interaction 
with the virtual can be more embodied and retain aspects of materiality and 
physical experience that virtual experiences through devices such as headsets 
or handheld tablets or phones could not replicate. 


Hybrid Objects 


Another approach is the creation of hybrid objects within which some aspects 
of the virtual and the material co-exist or are linked via real-time interaction. 
Bespoke hybrid objects have also crafted to suit period settings, such as historic 
houses, and to augment their atmosphere. For example, the Interactive Work- 
Table and Escritoire at Dr Johnson House museum in London (Patel et al. 
2015) was designed to aesthetically resonate with the house, while offering visi- 
tors a novel interactive experience. 

Another example are the smart replica objects designed for the historical 
exhibition The Hague and the Atlantic Wall held at Museon (The Netherlands). 
In this case, the hybrid replicas were small objects that could be carried in 
one’s hand, and each object was the replica of an authentic museum artefact on 
display and corresponded to a different theme of virtual content (visual and 
auditory) to be unlocked during the visit (Marshall et al. 2016) (Figure 13). 

In the Interactive Tableaux installation at the Bishops’ House Museum in 
Sheffield (UK), replica objects were also used, but with an added layer of hybrid 
interactivity. They activated a set of diorama-like tableaux representing different 
historical periods in the life of the house and its inhabitants. Every tableau reacted 
with different behaviours when activated: from playing a sound or light to emit- 
ting a smoky smell from its frame, displaying a video on a miniature screen inside 
the diorama, or making an automaton inside the diorama move (Claisse et al. 
2020) (Figure 14). Evaluation of the installation shows that visitors enjoyed the 
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Figure 13: The smart replica objects for The Hague and the Atlantic Wall at 
Museon (inset). Each replica reproduced an authentic object in the exhibi- 
tion (inside glass case in photo). Replicas of each object could be picked up 
by visitors (left of exhibition stand in photo) to explore the exhibition and 
trigger virtual content representing different points of view. Photographs by 

Daniela Petrelli and Nick Dulake, used with permission. 


Figure 14: The Interactive Tableaux at the Bishops’ House Museum. Each tab- 
leau reacted with different behaviours when activated with replica objects. 
Photographs by Caroline Claisse, used with permission. 
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multisensory aspects of the experience because they resonated with physically 
exploring a fascinating and atmospheric ancient house (Claisse et al. 2018). 
Hybrid objects can also be crafted for educational hands-on installations. 
For example, one exhibition by Maquil et al. (2017) for the Tudor Museum in 
Luxembourg resembled a scientists’ workbench where visitors learned how a 
battery can be built. They could choose and assemble components such as elec- 
trolytes, plates, and active paste, test the battery by revving a handle, and view 
its performance on a simulated graph paper displayed on an embedded screen. 


Virtual-Physical Assembly 


In this approach, the experience of virtual behaviours and content is designed 
side by side with the physical/material one, within the time frame of the visit. 
In this case, the assembly (as defined by Fraser et al. 2003) is a blend of vir- 
tual elements and material ones, including portable objects that are low-tech 
and do not offer any virtual experience per se, but that become part of one at 
some point by virtue of an underlying narrative of interactivity. These compo- 
nents are all linked together by a unifying activity that follows the narrative and 
introduces virtual elements at various points. 

One example is The History Hunt at Nottingham Castle (UK) (Fraser et al. 
2004), where low-tech paper worksheets were used to collect and file clues in 
the grounds of the castle and were subsequently augmented with RFID tags to 
activate mixed reality exhibitions in the castle gatehouse. Another example is 
Reminisce (Ciolfi & McLoughlin 2017); in this case, visitors exploring an open- 
air museum could collect ‘tangible tokens’ - small packages containing mean- 
ingful objects as well as digital audio snippets representing each of the historic 
cottages they visited on the trail. In the final building of the trail, the tangible 
tokens became activators for a separate installation concluding the visit. 


Hybrid Takeaways 


In this approach, material or virtual artefacts relating to an exhibition or 
site are available to visitors pre- or post-visit as hybrid takeaways. Therefore, the 
strategy is to realise a blending of the virtual heritage experience with a relevant 
material component that features either before or after the visit. Time becomes 
an important variable in this approach, as the hybrid experience becomes fully 
realised beyond the frame of the actual visit. How the takeaways are designed 
and their degree of digitality varies. One example is The Chantry, a freely 
available VR ambient game that is intended to be played before or after a visit 
to Dr Jenners House museum (UK). The games environment is a 3D model of 
Dr Jenner's House; however, the game is not intended to be played while physi- 
cally visiting the site but as a companion experience to be enjoyed at a different 
moment and possibly encourage repeated visits (http://revealvr.eu/2018/09/17 
/the-chantry-launched-on-european-playstation-store/). 
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Figure 15: The souvenir postcards (front and back) realised by the meSch 
project for Museon (left) and the Italian Historical War Museum (right), 
rematerializing virtual aspects of individual visits. Photograph by Luigina 
Ciolfi. Postcard design by Paddy McEntaggart for meSch. 


Conversely, the hybrid takeaway can be a physical token or object that is 
related to a virtual experience in some way. Instances of this are personal sou- 
venirs that re-materialise virtual or intangible aspects of a visit. For example, 
Nissen et al. (2014) facilitated the making of takeaway objects by visitors them- 
selves. The objects creatively represented what they had seen at a digital art 
exhibition. As part of the EU meSch project, takeaways took the form of per- 
sonalised postcards that were automatically generated to represent the experi- 
ence of virtual content that each visitor had, through the use of log data (Petrelli 
et al. 2017; Not et al. 2017) (Figure 15). 

One of these postcards would be printed with different ‘stamps’ correspond- 
ing to virtual heritage installations that were interacted with in a large exhibi- 
tion (Figure 15, left), or contain a written summary of the digital audio that a 
visitor had listened to at various points (Figure 15, right). In both these exam- 
ples, the postcards also displayed unique URLs that each person could use to 
access further digital content through an online visit, leading the way for more 
interaction with virtual content. 


Conclusion: Contextualising Virtual Heritage 


These approaches to hybrid design, and the examples provided to illustrate 
them, show that there is a wide range of possibilities to blend the virtual and 
the material in interactive installations at museums and heritage sites. Indeed, 
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we can argue that no interactive installation is ever fully virtual, simply given 
the fact that it is approached and experienced by people who have bodies 
and sensory capacities and who need to manipulate some kind of device in 
order to activate it. There is, therefore, no opposing dichotomy of virtual ver- 
sus material; rather, they exist on a continuum of possible embodied experi- 
ences of digital and indeed virtual heritage, with varying degrees of overlap 
and interrelationship. 

Practical instances of designing for human-computer interactions in 
museums demonstrate that there is no ‘one size fits all’ approach to realising 
pleasurable, effective, and meaningful virtual or hybrid heritage experiences 
(Hornecker & Ciolfi 2019). Therefore, understanding and thoughtfully 
responding to the heritage context is essential; a hybrid approach including 
elements of tangible interaction might be more relevant or feasible for certain 
types of museum or of heritage than others. Similarly, such contextual com- 
plexity should inform the decision of how the virtual should blend with the 
material and which instance of interactivity should be offered. Furthermore, 
the interpretation strategy of an institution is also an important factor. An insti- 
tution’s emphasis could be on historical aspects, or material culture, design, and 
crafting, or on intangible yet materially and bodily experienced aspects of folk 
traditions and oral history. 

All these considerations also weigh on the approaches to evaluating hybrid 
experiences: for example, whether the focus of evaluation should be on the 
educational aspect of exhibits or on the empathic experience of encounter- 
ing and appreciating traditions and cultures. Overall, the evaluation of hybrid 
experiences almost always puts a strong emphasis on their evocative nature, 
striving to document felt aspects of the visitor experience, as well as other more 
traditional indicators of memorability, flow, and learnability (Damala et al. 
2016). Qualitative methods are very commonly used to capture emotional 
and embodied aspects of engagement, and naturalistic studies allow for docu- 
menting and reflecting on these experiences in context (Ciolfi & McLoughlin 
2012; 2017). Furthermore, evaluation studies also might be concerned with the 
interweaving of the installations with the broader material context, from 
the hybridity and embeddedness of interactive behaviours in context 
(Hornecker 2010), to the aesthetic delight and surprise around hybrid exhibits 
(Taylor et al. 2015), and the environmental and atmospheric setting of hybrid 
interactions (McGookin et al. 2017). 

As digital technologies and platforms become more and more powerful and 
cheap and are more pervasively used in museums, awareness of their material, 
cultural, and organisational fit with heritage institutions is even more para- 
mount. Approaches to designing and evaluating digital interpretation, there- 
fore, might need to align to these strategies, meaning different roles for virtual 
or tangible instances of interactivity and for any possible blend thereof. 

In conclusion, this chapter particularly argues for the need to consider the 
key role that material (physical and tangible) facets of experience play when 
people approach museums, heritage sites, and heritage artefacts. While this is 
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acknowledged by numerous heritage studies experts and practitioners, inter- 
pretation strategies often neglect these aspects, and so do digital designs. Due 
to the fundamental embodied, sensory, and embedded elements (as well as 
the emotional and intellectual ones) of the human experience of technology 
(McCarthy & Wright 2004), the practice of designing encounters with virtual 
heritage must be cognizant of this complexity. 
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More and more, people do not experience the past through books, museums, 
or even television, but through video games. This chapter discusses how these 
popular entertainment products provide playful and fun experiences of the 
past - something we refer to here as past-play for the sake of brevity. The video 
game industry has become a major, fast-moving player when it comes to creat- 
ing, innovating, and distributing virtual representations of the past (Champion 
2015). The study of such playful video game-based products as examples of 
virtual heritage is part of a growing field, called archaeogaming. Archaeogam- 
ing can be generally defined as ‘the archaeology of digital games, with roots in 
a diverse set of analogue and digital archaeological themes and tools (Reinhard 
2018). It also draws in a variety of tools and thinking from game studies, game 
user research, and computer sciences. Archaeogaming is also a movement born 
in and out of playful, digital scholarship that studies popular, digital culture but 
itself also seeks to be part of popular, digital culture (Politopoulos et al. 2019a). 
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In contrast to other research in the field of virtual heritage, questions of accu- 
racy of reconstruction and scholarly standards are of secondary importance 
in archaeogaming - although the question Ts this historically accurate?’ fre- 
quently serves as a point of departure for many entering the field. The reason 
for this is that video games are not primarily (or at all) meant to be accurate 
reproductions of the past. Instead, they are first and foremost entertainment 
and artistic products that are meant to be fun. So, instead of asking 'How accu- 
rate is this digital recreation of the past in this game?’ game makers and most 
players will ask “How much fun am I having with (this part of) the game?’ An 
archaeogaming type of question can then be: “How and for whom is this part of 
the game, which is a digital recreation ofthe past, fun?' One answer may be that 
it is fun because it is an accurate, solidly researched representation of a thing 
from the past, but this is only one of the possible explanations. Other answers 
may be that heritage in a game provides a fun challenge (e.g., it is something tall 
to climb), is emotionally affecting (because it tells an upbeat or dark history), 
engaging, and relatable (because the player can personally relate to a character/ 
event/place), or makes you feel like you have gone back to the past in an actual 
time machine (because the game is interactive, immersive, and authentic). 

Fun may not seem to offer particularly solid ground for a serious line of 
research, but do not be deceived by some conservative yet influential voices 
that tell you that making, having, or thinking about fun is not serious or pro- 
ductive. Fun is a highly intriguing and complex thing; we all know it when 
we are having it, but it is difficult to explain what it is exactly or to recreate it 
consistently. We often use the definition of fun as posited by cultural philoso- 
pher and game developer Ian Bogost: fun as a relation of commitment, atten- 
tion, and care. The things you end up having these types of relations with is 
up to you to establish. However, having fun with (video) games is something 
that for a variety of conservative historical and economic reasons, go beyond 
the scope of this chapter (see Sharpe and Thomas 2019) but is somewhat of a 
taboo in academic circles. This is one of the reasons that a lot of archaeogaming 
work is done outside of traditional scholarly channels such as books and jour- 
nal papers, but through social media, blogs, live-streaming, and video essays. 
No matter in which form it is presented, work on how video games let us create 
and experience the past in ways that are fun as well as personally and societally 
impactful is as intriguing, complex, and important as any theme in (virtual) 
heritage studies — although, we acknowledge, it is perhaps a bit more fun. 

After providing some context to how past-play works and what forms it 
takes, we will review four different cases of past-play: the Assassins Creed series, 
an action-adventure series set in various periods and regions of the past; Never 
Alone, a game that turned into a global success, co-developed by the Ifupiat, an 
Alaskan indigenous tribe; RoMeincraft, a project developed to raise awareness 
of Roman heritage in the Netherlands; and the No Mans Sky Archaeological 
Survey, a project developed to document the heritage of a procedurally gener- 
ated galaxy with thousands of players. In each of these short reviews, we will 
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answer four key questions related to heritage in games: what sort of heritages 
do we experience through video games, who has created these, for whom, and 
how are they fun? These will provide concrete examples of the potential 
and pitfalls of the diverse ways we play with the past in the present. 


How Does Past-Play Happen? 


Often described as ‘magic circles, a term coined by the historian J. Huizinga in 
his seminal Homo Ludens (1938), games are frequently cast as things that mimic 
the 'actual world. This definition of games is helpful to get a first understand- 
ing of how they provide experiences of the past: by creating a ‘playground’ set 
aside in space and time from the here and now (Politopoulos et al. 20192). Even 
so, an expanding body of game studies has convincingly argued that a view of 
games as disconnected ‘magic circles’ minimizes the many ways in which this 
medium enduringly affects us as individuals and members of local to global 
communities, or how games themselves are entangled with ‘outside’ political, 
economic, social, and ideological forces (e.g., Keogh 2018; Dyer-Witheford & 
De Peuter 2009). 

Video games are rooted in and intertwined with a complex set of techno- 
logical, cultural, and societal factors. Firstly, video games are made possible 
through the processing of graphics, sounds, feedback, rules, and stories by 
computers (Newman 2010). Secondly, and as discussed above, they are gener- 
ally speaking meant to be for fun. To begin to understand how they do what 
they do, it is useful to view games as a blend of mechanics (game rules and 
computer programming), dynamics (gameplay and goals), and (visual, aural, 
haptic, and emotional) aesthetic components that work together to create com- 
pelling, immersive, and affective experiences. Players do not passively consume 
these components but interact with them by providing input to a computer; 
in doing so games frame their players as powerful, creative, and performa- 
tive agents. Player input is reacted to and evaluated by the computer or, in the 
case of social games, the computer and other players (Karhulahti 2015). 

It is important to realize that games are designed and crafted using a range 
of digital specialties, often distributed among teams of tens, hundreds, or even 
thousands of highly skilled people. The production and consumption of their 
work constitutes a growing, global, hi-tech, and competitive entertainment 
industry, on par with or even eclipsing movies, books, and other media indus- 
tries. While public history and heritage studies, as well as art history and media 
studies, have looked quite extensively at how the past is negotiated in all forms 
of media, comparatively little has been said about how video games function 
in this capacity. Studies of a subset of games, so-called serious games made for 
history and heritage outreach, provide an exception to this. Yet, according to 
recent overviews of this field, the picture is not clear and more research on their 
effects is badly needed (Champion 2016; Koutsabasis 2017). Moreover, serious 
games that are specifically designed to enable understanding of and access to 
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Figure 16: A network showing three ‘families’ of historical games, using data 
from Steam. Nodes represent game elements (‘tags’) that are identified and 
reported by the players of these games (produced by Angus Mol). 


the past are certainly not the most impactful forms of past-play, at least in the 
quantitative sense. Instead, the vast majority of past-play takes place in popular 
games like Civilization, Assassin’s Creed, and Battlefield. 


What Forms of Past-Play Are There? 


An estimated two billion people, or more, play games (Statista 2020). As a 
growing, global, and diverse community playing a large variety of game types, 
it is impossible to tell how many of those billions frequently play games set in 
the past and what sort of heritage and other experiences these games offer to 
them. Still, it is possible to get some sense of this by looking at what players 
themselves report on what they play. 

We did this by using a web-scraper - a little ‘robot’ or computer script that 
collects information from specific pages on the web - that collects information 
on historical games on Steam, a leading game online marketplace (Mol 2019; 
Figure 16). Steam allows players to tag, from over 300 different types of descrip- 
tors, the games they play, which it uses to provide recommendations to all of its 
users. Based on the analysis of game contents tagged by millions of players, it 
becomes clear that most historical games build on a relatively narrow set of ele- 
ments, leaning heavily on a view of history as conflict-driven (e.g., ‘Action and 
"War are commonly tagged game contents). While these games share many 
aspects, the study also showed they can be divided into three ‘families’: strategy 
games, action-adventure games, and first-person shooters (Figure 16). 
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User and sales numbers reported by industry members can provide a first 
insight into the popularity of past-play. For example, Firaxis, the creator of 
Civilization, stated that from 2010 to 2016 people collectively played more than 
1.2 billion hours of this game - eclipsing, for example, time spent in some of 
the world's best-visited museums (Mol et al. 2017). The maker of the Assassin's 
Creed series, Ubisoft, has shared that since 2007 these games have sold more 
than 140 million copies, making it one of the largest game franchises of the last 
decade. The publisher Electronic Arts stated, in a 2017 report to its sharehold- 
ers, that Battlefield 1, a 2016 game set in WWI, had attracted 25 million players 
in a little over a year (Minotti 2018). 

Given these numbers, it seems difficult to overstate the popularity of past- 
play, yet high sales or engagement does not necessarily equate to cultural or 
societal impact. Yet some evidence already supports the idea that these popular 
games have a significant influence on how we relate to and think about the past, 
amongst which a survey of players of popular historical games recently under- 
taken by us and two of our students, Omar Bugter and Stefan Tibboel (Mol et al. 
2020). For example, 93% of respondents to the survey (n = 1670) reported that 
games have inspired them to learn more about a historical person or event and 
90% agree with the statement that video games can change peoples viewpoints 
on a historical event. 

Notwithstanding the deep interest individual game makers may have for 
history and heritage, popular historical games are traditionally and still fre- 
quently pitched as ‘magic circles. They offer forms of digital heritage that — for 
the sake of fun and finances and in the opinion of those in control - should 
be seen as separate from the cultural and societal dynamics of the 'outside 
world. For example, Sid Meier, lauded game designer and creator of Sid Meier 
Civilization, has stated that 'one of our fundamental goals [in making Civiliza- 
tion] was not to project our own philosophy or politics onto things. Playing 
out somebody else's political philosophy is not fun for the player? A. Condelius 
(in Taylor 2018), COO of Ubisoft, the developer of the Assassins Creed series, 
has similarly stated that ‘People like to put politics into [our games], and we 
back away from those interpretations as much as we can [as they are] bad for 
business, unfortunately: 

Yet many smaller studios and independent creators do find new ways to pro- 
vide players with an engaged, possibly fuller, sense of histories and heritages. 
Some of these so-called 'indie titles also have been met with huge critical and 
commercial success. In addition, once a game is in the hands of its players, 
virtual heritage in it can take on all sorts of different shapes not intended by the 
developer - for example through the things players make in-game or through 
extensive modifications, ‘mods; to the game. This provides a space for crea- 
tive and surprising engagements with the past and makes new forms of virtual 
heritage. It is therefore important that any prospective archaeogamer has an 
eye for virtual heritage in a variety of forms: as produced and consumed in 


86 Virtual Heritage 


a == 


The front of the Athenian 


Athenian Treasury at Delphi P 
Treasury 


The remains of the 6t Temple 
of Apollo at Delphi 


The Classical temple in 

Assassin's Creed Odyssey 

Figure 17: Buildings at Delphi in AC: Odyssey and the same locations in real- 
ity (after Politopoulos et al. 2019). 


‘blockbuster’ games such as Assassins Creed, through the innovative interven- 
tions of ‘indies, and as created by the many people who play with the past. 


Assassin's Creed 


The Assassins Creed series (AC) is one of the most popular and enduring video 
game series set in the past. Twenty-one AC games and spin-offs have been 
released since its initial appearance in 2007, across several platforms, all using 
various historical settings and characters. Eleven games belong to the core set 
of releases and can be described as third-person action-adventure games. The 
action-adventure aspect of the game revolves around three main mechanics: 
fighting, stealth, and parkour. However, as it is evident from the title of the 
series, killing is central to how the stories of the games play out. 

The backstory of the game presents two shadowy organizations, the Assas- 
sins and the Templars, which fight for the acquisition of ancient artefacts and 
technologies belonging to a precursor advanced civilization. One of these tech- 
nologies, the Animus, allows individuals to visit the past through memories of 
their ancestors. Using this as a core narrative device AC allows the player to 
visit multiple periods and places of the past. This broad theme, which gives an 
opportunity to the developers to revisit pretty much any historical (or fictional) 
location and time, combined with a detailed and attentive reconstruction of 
past places, has been central to the success of the series. This is exemplified 
in the highly detailed reconstruction of Athens or Delphi in AC: Odyssey, the 
latest addition to the series (Politopoulos et al. 2019b; Figure 17). 
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All the games are developed by Ubisoft, one of the largest video game com- 
panies globally, with development studios around the world. Different develop- 
ment studios have been responsible for the various releases, providing a form 
of diversity in the development process; however, one would be hard-pressed to 
argue that AC games have been considerably different from each other, despite 
the diversity of settings. In that lies both the success, but also the pitfall of the 
series. AC games offer a widely popular yet relatively repetitive approach to 
both gameplay and the past. While the player gets to experience beautiful set- 
tings and meet historical characters from several periods, this experience can 
only be had through killing. In the end, the game creates a weird balance of 
killing and heritage tour, making the past an enticing yet violent place to be. 


Never Alone 


In Never Alone, you play as the Ifupiaq girl Nuna and Fox, who seek to find 
the source of a blizzard that has destroyed her village. The two-dimensional 
game marries platforming (i.e., jumping and running) with puzzles that can 
only be solved when Nuna and Fox work together. As a player you will encoun- 
ter all sorts of obstacles that are part of the cultural and natural world of north- 
ern Alaskan, from being chased by a polar bear to having your head taken off 
by Northern Lights Spirits. The game is based on the living history and heritage 
of the Iñupiat (the plural for Iñupiaq), an Alaskan indigenous group. One of 
the games mechanics is ‘Cultural Insights, collectibles that let the players watch 
video interviews with Ifupiat in which they tell about all sorts of their life and 
histories with information that can be directly related to the puzzles, enemies, 
or items you as a player are encountering. 

The game is the result of a co-creative process between The Cook Inlet Tribal 
Council (CITC), a tribal non-profit serving the Ifupiat people, and E-Line 
Media, a game developer originally based in Seattle. Looking for something 
that would be a way that would allow them to keep their traditional culture 
alive but also speak to the younger members of their tribe, the CITC decided 
to invest in game development. The story of this development of Never Alone — 
which involved a first visit by E-Line in the middle of a January blizzard, get- 
ting the permission of the person who was the owner of the main story told in 
Never Alone, the creation of a unique art style based on Iñupiaq art, the deci- 
sion to narrate the game in Iñupiaq - is an enormous success (for the rest of this 
fun history, see CITC 2016). Not only did the game manage to teach Ifupiaq 
values and tradition to their children, but it also became a commercial and 
critical success. 

The game is beautiful, and the Cultural Insights are enthralling, providing 
a window for non- Iüupiat into a world they may never visit but will come 
to respect or even love through play. Perhaps more importantly, it is also fun, 
something that is achieved through the cooperative puzzle-solving between 
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Nuna and her Fox friend. As is explained in the first Cultural Insight, the idea 
that we are never alone is a key part of Ifupiaq worldview - the alternative title 
is Kisima Inyitchuya, Ihupiaq for ‘I am not alone. Making connections with 
others through cooperative or social gaming is a game activity that is found fun 
by many. The human connection at the heart of Never Alone is a great example 
of how mechanics and aesthetics vested in a cultures most cherished heritage, 
as well as a key part of the culture of gaming, come together to create some- 
thing truly special and fun. 


RoMeincraft 


RoMeincraft is a project developed and carried out by the VALUE Foundation, 
of which the authors are founding members. The projects aim was to raise 
awareness of existing or lost Roman heritage in the Netherlands, and particu- 
larly the Dutch Limes, through the reconstruction of Roman cultural heritage 
sites in the popular video game Minecraft, together with the broader public 
(Politopoulos et al. 20192). 

Minecraft is best described as the LEGO of the video gaming world. It is an 
open world sandbox game without a real purpose but to build whatever the 
player likes. The Minecraft world consists of pixelated blocks of various materi- 
als that the player can dig up, collect, and use to construct pretty much any- 
thing. The game is offered in two modes: the survival mode, where the player is 
thrown in a randomly generated map and has to survive by any means neces- 
sary, and creative, where the player has limitless access to every material in the 
game. This latter mode is what made Minecraft ideal for our project. 

For RoMeincraft, we designed scaled maps of the Netherlands (ranging 
between 1:2-1:4), with the locations of Roman forts being 1:1. In Minecraft 
every block is 1x1x1m, which makes it easier to measure space. We then con- 
ducted a number of public events in museums, plazas, and cafés, where our 
visitors could use one ofthe provided computers (or even alaptop they brought 
themselves) to go to a given location and start reconstruction of Roman forts. 
We provided players with supplementary material with historical and archaeo- 
logical information about how these forts looked, but generally, players were 
free to construct anything they imagined would exist in Roman Netherlands. 
This resulted in a number of impressive constructions, several ancient Greek 
temples, as well as a polar bear infestation (Figure 18). 

The RoMeincraft project and other projects like it show that a game does not 
necessarily need to be set in the past to still become an object of virtual heritage 
study and outreach. It is, rather, the use we make of them that has the potential 
to create new and engaging ways of presenting virtual heritage to a wider audi- 
ence. The open and creative nature ofthe project gave an opportunity to visitors 
to engage with Roman heritage in a new way, but also in their own way, either 
by building in Minecraft, watching other people build, discussing, reading, 
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Figure 18: Screenshot of the reconstructed fort Matilo in Minecraft that had 
been built over several RoMeincraft events held in Leiden, 2017 (Authors). 


or simply by being there and experiencing the event. This is an example of what 
happens when fun, rather than historical or graphical fidelity, is the aim of vir- 
tual heritage projects. 


The No Mans Sky Archaeological Survey 


The No Mans Sky Archaeological Survey (NMSAS) is arguably the most 
strictly archaeological project undertaken within a video game (Reinhard 2018: 
88-160). Conducted by Andrew Reinhard as part of his PhD study, its aim 
was to demonstrate the viability and validity of archaeological methods for the 
study of digital spaces. 

No Mans Sky is an online multiplayer exploration-survival game developed 
by Hello Games. It is set in a procedurally generated galaxy, which spawns an 
infinite amount of algorithmically generated planets. The player is dropped in 
a random location of the galaxy and has to work their way to its centre, all 
while visiting planets to extract resources that can either be sold or used to 
construct or buy equipment and improvements for the player's spaceship. The 
planets, however, not only contain natural landscapes but also randomly gener- 
ated ruins of extinct cultures or even remains of players’ settlements that have 
been ruined by software updates. 

Reinhard implemented a full-scale community-driven survey to record all of 
this in a systematic manner based on the archaeological methodology of field 
survey. This project aimed to record now-extinct player and algorithmic herit- 
ages, as well as secure information on existing ones, before another catastrophic 
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event would wipe them out. The NMSAS was opened up to Twitter and the 
Reddit community ofthe game, which allowed Reinhard to collect significantly 
more data than any given survey projects can realistically collect, through play- 
ing archaeology in a virtual, player-created world. This demonstrated both the 
importance but also the possibilities archaeogaming offers in doing research 
together with a wider audience as well as how it can research at larger scales 
and using archaeological frameworks, virtual heritage that is itself born digital. 
The NMSAS was both fun for its participants and offered new ways of thinking 
about digital spaces and heritage. 


Conclusion 


While many of the qualities of video games overlap with other (popular) media, 
the interplay of all of these - computational, fun, immersive, affective, interac- 
tive, performative, crafted, industrial, ‘magic, entangled - means that games 
provide one-of-a-kind experiences, not only in the present but also of the past. 
The examples presented above highlight only some of the potential virtual her- 
itage experiences produced through video games. It is exactly this combination 
of qualities alongside the wide diversity of past experiences that make video 
games stand out from other popular media. Video games are useful not (only) 
because they can offer us accurate representations of the past; in fact, one could 
even say that this is the least interesting of their potential uses. Rather, video 
games are a crucial form of digital heritage because they can offer us concepts of 
the past, ways of thinking in, around, and about the past, as well as experiences 
of the past through play. 

Assassins Creed games give players an opportunity to walk around cities of 
the past, and talk and interact with historical figures that you only read about in 
books. Games such as Never Alone, produced by indigenous populations, allow 
developers to safeguard cultures and players to encounter new ones virtually, 
and in the process provide access to concepts of the past that would otherwise 
remain inaccessible. The RoMeincraft project illustrates the space video games 
offer for promoting digital heritage, not only as educational but perhaps more 
so as pure play. This also underlines the co-creative and co-mediated side of 
heritage outreach. Finally, the No Mans Sky Archaeological Survey shows that 
heritage is not only something that can be put in video games but can also be 
born virtual. This requires new ways of studying these virtual cultures, as well 
as new room to play around with the concept of recent, fleeting, and born- 
digital pasts. 

The common denominator, however, in all these experiences is fun. These 
games not only offer diverse approaches to share, study, and create virtual 
heritage through past-play past, but also as a relation built on commitment, 
attention, and care - in other words as something fun. Work in archaeogam- 
ing and other forms of past-play has already shown some of the potential of 
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games but the breadth of experiences that can be had in these games as virtual 
heritages shows that there is much more to be done and much more fun to 
be had. 
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Abstract 


Virtual heritage (VH) is one of the few domains to adopt immersive reality 
technologies at early stages, with a significant number of studies employing 
the technologies for various application themes. More specifically, virtual real- 
ity has persisted as a de facto immersive reality technology for virtual recon- 
struction and virtual museums. In recent years, however, mixed reality (MxR) 
has attracted attention from the VH community following the introduction 
of new devices, such as Microsoft HoloLens, to the technological landscape of 
immersive reality. Two variant perceptions of MxR have been observed in the 
literature over the past two decades. First, MxR is perceived as an umbrella/ 
collective term for a virtual reality (VR) and augmented reality (AR) environ- 
ment. Second, it is also presented as a distinctive form of immersive reality 
that enables merging virtual elements with their real-world counterparts. These 
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perceptions influence our choice of immersive reality technology, interaction 
design, and implementation, and the overall objective of VH applications. 

To address these concerns, this chapter attempts to answer two critical ques- 
tions: (1) what MxR from VH perspective is and (2) whether MxR is just a 
form of immersive reality that serves as a bridge to connect the real world with 
a virtual one or a fusion of both that neither the real nor the virtual world 
would have meaning without a contextual relationship and interaction with 
each other. 

To this end, this chapter will review VH applications and literature from 
the past few years and identify how M3R is presented. It will also suggest how the 
VH community can benefit from MxR and discuss limitations in existing tech- 
nology and identify some areas and direction for future research in the domain. 


Introduction 


Despite the significant advancements observed in the technological landscape 
of immersive reality and its expanding applicability across various domains, the 
perceptions of immersive reality technologies in general or at least their depic- 
tion in the VH literature remains influenced by earlier theoretical and techno- 
logical perspectives - missing current contextual and domain-specific views. 
For instance, one of the earliest and widely accepted definitions of augmented 
reality (AR) by Azuma (1997), a segment of the reality-virtuality continuum 
proposed by Milgram and Kishino (1994), depicts AR as ‘a system that com- 
bines real and virtual content, provides a real-time interactive environment, 
and registers in 3D? 

In addition to AR being presented as a system/technology, the character- 
istics that identify the segment from the rest of the continuum are that it 
‘combines real and virtual’ content and ‘provides real-time and 3D interactive 
environment. These properties are observed similarly in MxR systems and envi- 
ronments, making AR and MxR identical or interchangeable as they attempt to 
combine real and virtual content and provide 3D interactive environments. As 
such, distinguishing AR from MxR relying on such properties is difficult. One 
of the primary objectives of this chapter is, therefore, to delineate a boundary 
between AR and MxR, at least from the VH point of view (the assumption 
is that the boundary between MxR and VR is much clearer as much as it is 
between AR and VR). To this end, establishing the current depiction of AR and 
MxR in the literature is required. Furthermore, distinguishing MxR from the 
rest of the segment requires identifying key factors from the VH perspective. 

To date, there are two widely conveyed definitions of MxR in the literature. 
First, MxR is perceived as a combination of AR and VR. For instance, Elrawi 
(2017), Makino and Yamamoto (2018), and Plecher et al. (2019) present MxR 
as a combination of AR and VR environment and/or a collective term repre- 
senting both AR and VR. This has led to the consideration of AR and VR as 
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the primary platforms for highly immersive and interactive VH applications 
(Haydar et al. 2011; Papagiannakis et al. 2018). Further to this, the technical 
complexity and requirements of fusing real and virtual elements, which is a 
unique property of MxR, to the extent that the blended environment appears 
as real as the real world has remained extremely challenging. This has to 
some extent resulted in a lower number of MxR applications and paved a favour- 
able path for AR's and VR5 position as the default platforms/technologies. 

Second, contrary to the first view, some studies consider MxR as a unique 
segment of the reality-virtuality continuum that is characteristically and tech- 
nologically different from both AR and VR. For instance, Jacobs and Loscos 
(2006), Okura et al. (2015), Bekele and Champion (2019b), and Hammady et al. 
(2020) present MxR as a technology and virtual environment that amalgamates 
real and virtual worlds into a single and shared real-virtual spectrum. 

Hence, it is evident that a common understanding of MxR is required before 
an attempt is made to answer the critical question ‘Is mixed reality a bridge 
between two worlds or a fusion of two worlds?’ 


Contextual Relationship in Augmented and Mixed Reality 


The widely accepted definitions of AR and MxR in the literature rely on systems 
and technological perspectives. Distinguishing MxR from AR and the rest of 
immersive reality technologies, therefore, requires identifying additional fac- 
tors from a different perspective rather than the underlying technology and 
theoretical basis. To this end, an article published by Bekele and Champion 
(2019b) identifies a contextual relationship between users, the real world, and 
the virtual environment as a factor that differentiates a specific form of immer- 
sive reality from the rest of the segments of the spectrum. 

The contextual relationship is realised when the combination/blend of the 
real and virtual environments enables a three-way interaction between users, 
reality, and virtuality. Establishing a contextual relationship also relies on 
how the blended environment resembles and feels as real as the real world. 
The outcome is an enhanced and engaging real-virtual space that ultimately 
allows users to establish a contextual relationship with the real-virtual environ- 
ment. The fusion and the three-way interaction are equally important factors 
to outline a boundary between AR and MxR. From a VH point of view, com- 
municating or obtaining meaning and cultural significance through immersive 
reality without a mechanism to establish such a contextual relationship will be 
a difficult task. Considering fusion and contextual relationship as additional 
differentiating factors, AR and MxR can be outlined as follows. 

Augmented reality is a form of immersive reality that enhances our percep- 
tion of the real world and allows users to interact with reality and virtuality. 
Usually, virtual content is superimposed onto our view of the real world. The 
content could be in any multimedia format ranging from text to 3D models. 
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Figure 19: Augmented reality is a form of immersive reality that enhances our 
perception of the real world and allows users to interact with reality and 
virtuality (figure produced by the author). 


As a result, there is relatively less expectation of the real-virtual environment 
resembling the real-world. 

In addition to this, the resulting real-virtual space in AR does not allow a 
three-way interaction between users, reality, and virtuality. Users are usually at 
the centre of the interaction establishing a direct relationship with the real world 
and the virtual environment. For instance, digital content (text, video, audio, 3D 
models) of cultural heritage assets can be superimposed over our view of the 
real world. In some cases, such as virtual reconstruction, digital content can be 
superimposed on top, or projected next to the same heritage assets in the real 
world. In this scenario, the virtual environment that is visible to users through 
AR technology relies on the assets in the real-world to communicate the com- 
plete meaning of the multimedia content. The physical assets in the real world 
would have meaning on their own but users’ understanding of the assets’ cultural 
significance would be enhanced with the AR technology. Figure 19 presents AR 
as immersive reality technology that allows users to interact with a real-virtual 
environment, enables a contextual relationship between users and the real- 
virtual environment, and enhances the users' understanding of the real world. 

Mixed reality, on the other hand, is a distinctive form of immersive reality 
that enhances our perception of both the real and virtual environments and 
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Figure 20: Mixed reality is a form of immersive reality that enhances our 
perception of both the real and virtual environments and allows interaction 
between users, reality, and virtuality (figure produced by the author). 


allows interaction between users, reality, and virtuality. Figure 20 presents MxR 
as immersive reality technology that allows users to interact with a real-virtual 
environment, enables a three-way contextual relationship between users, the 
real world, and the virtual environment, and enhances users’ understanding of 
both the real world and the virtual environment. 

The real-virtual environment (a combination of real and virtual) provides 
a shared space that elements from both worlds utilise to enhance our under- 
standing of both worlds. In this regard, the difference between AR and MxR is 
that the virtual environment in AR is limited to enhancing our understanding 
of the real world. Hence, the relationship between the real and the virtual envi- 
ronment in AR is limited to a one-way direction. The virtual environment in 
MxR, however, is not limited to enhancing the real world. It also benefits from 
the real word for delivering enhanced meaning. This arrangement results in 
a three-way relationship between users, reality, and virtuality. 

For instance, consider shipwrecks or physically recreated replica of ships in a 
museum. Conveying the history and cultural significance of the ships to visitors 
can be realised via AR (superimposing multimedia content and 3D models) 
or via MxR (blending virtually simulated 3D animated model of the crew 
and the physical recreation of ships). Both approaches can enhance visitors’ 
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understanding of the ships. However, the MxR approach provides a shared 
space for both the physical ship and the virtual simulation to communicate the 
complete picture of the story of the ship. Because, in this scenario, the simula- 
tion and the physical heritage asset are highly dependent on each other. 

In summary, VH can adopt multiple forms of immersive reality technology 
to achieve a similar objective (i.e., whether explicit or implicit, VH applica- 
tions tend to aim at communicating/transmitting the significance and value of 
heritage assets to visitors/users of the applications). However, considering the 
available technologies (AR, VR, AV, and MxR), a specific form of immersive 
reality can deliver the expected outcome more effectively than the rest. This 
is even more evident when comparing AR and MxR against their potential to 
enable a three-way contextual relationship between users, reality, and virtuality 
and blending the real and virtual environments. 

As Table 1 shows, MxR exhibits unique aspects especially in terms of estab- 
lishing a contextual relationship between reality and virtuality and blending the 
real and virtual environments to the extent the fusion is as realas the real world 
that results in benefiting both worlds. These unique features of MxR make the 
technology an ideal choice for VH applications that aim at virtuality recreating 
or simulating partially or completely lost tangible and intangible heritage assets 
and blending them with their counterparts that still exist in the real world. 


Table 1: Comparison of AR and MxR against their potential to enable a three- 
way contextual relationship between users, reality, and virtuality and blend- 
ing the real and virtual environments. 


Factor Augmented Reality Mixed Reality 
Blending the real Overlays virtual content | Virtual content is blended with 
and virtual onto the real world the real environment resulting in 
environments a shared real-virtual environment 
Interaction Users can interact and Users can interact and establish 
between usersand | establish a contextual a contextual relationship with the 
the real world relationship with the real world 

real world 
Interaction Users can interact and Users can interact and establish a 
between users establish a contextual contextual relationship with the 
and virtual relationship with the virtual environment 
environment virtual environment 
Interaction There is no interaction There is a continuous contextual 
between the real between the real relationship between the real 
world and the world and virtual world and virtual environment 
virtual environment | environment in ARand | in MxR to the extent that specific 
the sole purpose of the meaning (e.g., cultural signifi- 
virtual content is cance in VH) can only be derived 
enhancing the real world | from the relationship 
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Mixed Reality: Bridge versus Fusion 


Having the boundary between AR and MxR outlined, this section attempts to 
determine whether MxR is a bridge that connects the real and virtual world 
or a fusion of the two worlds that serves as a shared space where contextual 
relationship, collaboration, and engagement can be realised to a higher degree 
of realism. To answer this crucial question, we need to establish the aspects and 
scope of immersive reality as a bridge and immersive reality as a fusion from the 
context of VH and the objectives of this chapter. 

Immersive reality technology can serve as a bridge between two worlds 
connecting us to past and/or lost cultures and heritages. In the context of the 
applicability of immersive reality in VH, the 'two worlds' refer to the exist- 
ing physical world and a virtually simulated environment that is spatiotempo- 
rally distant from the existing physical world. The bridge analogy is, therefore, 
characterised as a spatiotemporal vehicle that can transport us to a different 
time and/or a different place. A typical immersive reality technology with such 
capability is VR. This technology can deliver a platform for highly immersive 
virtual environments that can simulate multiple dimensions of past traditions, 
cultures, and heritages. The immersivity of VR is not limited to the spatial and 
geometrical aspects of the simulated virtual environment. An ideal simulation 
will consist of multidimensional aspects of the simulated culture/heritage such 
as temporal, attributive, and environmental parameters. Such simulations can 
effectively transport us to the past to the extent that we are tricked to believe we 
are situated there and then. 

Alternatively, immersive reality technology can also fuse the real and virtual 
worlds. From a VH perspective, the fusion of the two worlds is a real- 
virtual environment that serves as a shared space for the past and the present 
to coexist (Brondi et al. 2016). Past cultures and civilisations can virtually reoc- 
cupy or blend with the existing physical environment. Unlike the bridge anal- 
ogy, which transports us to a past and distant world, the fusion of two worlds 
lets us experience the same past and distant world interacting with the exist- 
ing physical reality that surrounds us. The fusion, therefore, exhibits properties 
of both the real and virtual environments that ultimately enables a contextual 
relationship between the two worlds. 

All forms of immersive reality technologies except VR can blend real and 
virtual environments at different levels of interactivity, immersivity, and con- 
textual relationships between components. For instance, a properly designed 
and implemented augmented virtuality (AV) system can blend the real and vir- 
tual environments in real-time. In this case, a live scene from the real world is 
streamed into the virtual environment rather than cases of AR where the fusion 
results in virtual content augmenting the real world. With both AV and AR, 
there is always a dominance of one environment over the other. The third alter- 
native is an MxR technology where the fused real-virtual environment serves 
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as an equally shared space for both realities. However, technological advance- 
ment is far from a state that such fusion can be realised to its full extent. Con- 
sidering existing technologies, however, MxR is a typical form of immersive 
reality that is best suited for fusing the real and virtual environments. 

Relying on how MxR is outlined in the context of VH in this chapter, the 
environment in MxR is a fusion of two worlds rather than a bridge between two 
worlds. This is because: 


e MxR enables a contextual relationship between users, reality, and virtuality. 

e MxR provides a balanced and shared space for elements from both the real 
and virtual worlds to interact with each other. 

e Both the real and virtual worlds can be meaningful by themselves 
(unlike AR, where the virtual environment relies on the real world to be 
meaningful). 

e Both worlds depend on each other for enhanced meaning. 


Mixed Reality and Virtual Heritage 


A significant number of studies have demonstrated the role of immersive real- 
ity technology in terms of enriching cultural heritage sites and museums with 
engaging, interactive, and immersive experiences (Hammady et al. 2020). 
Recent technological advancements have made MxR even more beneficial and 
accessible to VH applications that tend to target virtual reconstruction in situ. 
Considering such recent development and trends, the followings have been 
identified in the literature as viable application themes of VH: 


1. Virtual reconstruction. Virtual reconstruction relates to the recrea- 
tion of fully or partially lost tangible or intangible cultural heritages. 
MxR is the best choice for VH applications with such themes because 
the technology can blend the reconstructed virtual environment with 
physical objects that exist at the historical location of the cultural 
heritage assets (Montagud et al. 2020). 

2. Virtual exploration. VH applications designed for virtual exploration 
aim at knowledge and insights discovery because of the VH applica- 
tion’s capability to afford manipulation and meaningful interaction 
with the underlying data and real-virtual environment (Okura et al. 
2015; Tennent et al. 2020). 

3. Virtual exhibition. Virtual exhibitions either replace physical muse- 
ums and heritage sites with simulations in VR or improve/enhance 
users’ experience at museums and heritage sites by blending virtual 
content with the real world, for instance, virtual tour guides in MxR 
(Trunfio & Campana 2020). 
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Virtual educational tools. To some extent, all the above applications 
serve as tools to educate/inform users regarding the historical and 
cultural aspects ofthe content presented in the applications. However, 
effective dissemination of cultural significance (cultural learning) 
requires VH applications that primarily focus on the outcome and 
learning aspects of the virtual content, application design, and imple- 
mentation of immersive reality. To this end, MxR is a viable choice as 
the technology enables engagement, interaction, and contextual rela- 
tionship with the real-virtual environment (key characteristics of VH 
applications that aim at cultural learning). 


Current Issues and Future Directions 


Mixed reality technology as it stands has several limitations hindering its wider 
adoption. The limitations identified in existing studies include rendering per- 
formance, lack of robust environmental tracking solutions, and a lack of easy- 
to-use multimodal interaction interface (Bekele 2019). Considering ongoing 
research on cloud-based immersive reality and human-computer-interaction 
(HCI), it is expected that future research will focus on the following areas: 


1; 


Cloud-based rendering. Rendering is perhaps one of the key techni- 
cal issues that MxR applications face across domains. It is even more 
problematic in VH applications that present sophisticated 3D mod- 
els with millions of polygons. Even the market-leading MxR device, 
Microsoft HoloLens, struggles to render 3D models with such a large 
number of polygons. As a result, decimation is required to reduce the 
number of polygons, which will then deduce details from the model 
impacting user experience and the vividness of the rendering. How- 
ever, Microsoft Azure announced a cloud-based remote rendering 
service as part of their MxR solutions. The remote rendering service 
will handle all the graphical computation workloads from the MxR 
device. Meaning, sophisticated 3D models can be rendered remotely 
and streamed to the MxR device, which is the Microsoft HoloLens. 

Cloud-based tracking. Sensor and camera-based tracking solutions are 
commonly adopted in existing VH applications. However, these solu- 
tions, particularly in outdoor settings, remain error-prone, impact- 
ing user experience. In this respect, new cloud-based services, such 
as Microsoft Spatial Anchor, provide the possibility of utilising cloud 
computing to store, share, and retrieve location data of points of inter- 
est for MxR applications across multiple platforms and devices. Mean- 
ing, VH applications can target multiple devices for user experience 
while maintaining a shared and centralised pose tracking solution. 
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3. Multimodal interaction interface. An ideal multimodal interaction 
interface combines multiple modes of interaction allowing users to 
interact with virtual environments as they would interact with the real 
world (Bekele & Champion 20192). This is a key property of MxR 
experience. Existing technologies rely on gaze, gesture, and speech 
inputs to enable multimodality in interaction interfaces. For instance, 
Microsoft HoloLens utilises all three inputs. As research advances 
in sensor technology, artificial intelligence, and tangible interaction, 
more advanced multimodal interaction interfaces will likely become 
a common method of interaction in VH, thereby enabling engaging, 
interactive virtual environments that users can effectively relate to and 
interact with through all their senses. 


Conclusion 


This chapter has presented different perceptions of MxR, especially in the VH 
domain. It has also outlined a boundary between AR and MxR before attempt- 
ing to answer the key question raised in the chapter 'Is MxR a bridge between 
two worlds or a fusion of two worlds?’ Immersive reality technology's capabil- 
ity to establish a contextual relationship between users, reality, and virtuality 
and believability and realism of the real-virtual environment resulting from the 
fusion of the real and virtual worlds were used as differentiating factors. I have 
identified application themes and limitations for MxR and VH applications as 
well as future research areas and directions that I invite you to explore. 
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Abstract 


Though the ethics of archaeological practice have changed over the life of the 
discipline (and have arguably become more robust), full consideration has 
not yet been given to how digital methodologies and the emergence of digital 
technologies have created new areas requiring ethical introspection. The pace 
of adoption of digitally centred archaeological data and digitally facilitated 
archaeological practice has not been met by the adoption of discipline-wide 
standards related to archaeological ethics. The result of this mismatch in eth- 
ics and practice is the creation of archaeologists who utilize digital forms, but 
whose archaeology is ungrounded in frameworks that specifically consider the 
ethical burdens of digital tools, methodology, and theory. This chapter details 
views of digital archaeological ethics related to digital archaeology as tools, 
digital archaeology as methodology, and digital archaeological pedagogy. 
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Introduction 


As archaeologists, we operate from an inherent place of privilege regarding our 
access to direct evidence of the past. We get to handle, study, and analyse arte- 
facts that for a variety of reasons, are not available to the public in the same way. 
Because of this privilege, we have what is called a ‘duty of care’ to act respon- 
sibly towards the sites we excavate, as well as to the public who relies on us to 
produce data that is accessible, understandable, and thoughtful. The chief way 
in which we consider whether we are meeting that responsibility is through our 
practice of ‘archaeological ethics: 

Archaeological ethics are the discipline-wide standards that archaeologists 
have agreed to uphold. They inform everything from how new archaeolo- 
gists are trained (Mills et al. 2008) to how excavation data is published (Kansa 
& Kansa 2013) to how we work with paying clients and the many different 
publics that are the ultimate recipients of our knowledge production (Fowler 
2017). However, these ethics are not static. As the discipline of archaeology has 
changed, so have the accepted ethical practices, as archaeologists have dealt 
with profound changes in the context of archaeology and profound changes in 
ethical concerns (Wylie 1996). 

Most recently, archaeologists have begun to consider the ethical implications 
of the digital in our work. This has come in many ways, two of which we will 
discuss in more depth. The first issue is how archaeological ethics should con- 
sider the digital tools that we use. The second issue is how archaeological eth- 
ics should consider the digital methodologies we employ. A critical, though 
as yet under-discussed, third issue is how we should consider archaeological 
education and the digital. (This third issue is what we sometimes call ‘digital 
archaeological pedagogy’) 

As a dedicated practice, digital archaeology is too recent to be included in 
the published guidelines and codes of ethics that are provided by organizations 
such as the Society for American Archaeology (SAA), the Society for Historical 
Archaeology (SHA), and the European Association of Archaeologists (EAA). 
This does not mean that digital archaeologists should operate without ethical 
oversight though! Digital archaeology has expanded into mainstream archaeol- 
ogy, and the ethics of practice of that expansion have just not been kept up with 
by the professional organizations in their documentation (Dennis 2020). 

One of the first discussions of what would result in ‘digital archaeology 
occurred in 1967, when Chenhall considered the electronic computer as a 
tool for data storage and retrieval (Chenhall 1967). Cowgill, also that year, dis- 
cussed the introduction of computers for statistical and computational analysis 
(Cowgill 1967), and between the two, the push to a digital archaeology had 
begun. A series of arguments for and against the use of computers in archae- 
ology occurred throughout the 1970s, and by the 1980s there was an explo- 
sion of computer-based archaeological data production. This is when total 
data stations and GIS began to enter archaeology. By 1992 the concept of a 
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digital archaeology and digital archaeologists had become prevalent enough to 
support the founding of the Computer Applications & Quantitative Methods 
in Archaeology organization (CAA), a professional group focused on digital 
archaeology. A digital archaeology interest group followed shortly after within 
the SAA. Despite all that, it was not until 2003 that the first direct mention of 
ethics in digital archaeology was published (Bayliss 2003). Even then, CAA did 
not have a dedicated ethics policy for digital archaeology until 2018, and SAA 
and EAA still do not, though both organizations are in the process of updating 
their codes of ethics more broadly as of 2021. 


Current Archaeological Ethics 


There are two circumstances under which most archaeologists encounter for- 
mal processes of archaeological ethics. The first circumstance is through the 
need to comply with mandated ethics frameworks, usually provided by univer- 
sities and research bodies who grant funding. These organizations ask archae- 
ologists, both as student researchers and as faculty researchers, to detail how 
were going to undertake our projects, and how were considering ethics in 
those projects. 

The second circumstance is less common, but is on the rise, and occurs when 
archaeologists meet at professional conferences or submit papers for publi- 
cation. Professional organizations, like the previously mentioned SAA, SHA, 
EAA, and CAA groups, ask members to follow what are known as ‘aspirational’ 
codes of ethics. 

Aspirational codes of ethics are intentionally general and are meant to apply 
to the majority of archaeological projects that follow the traditional physical 
excavation and journal publication model of archaeology. Aspirational codes of 
ethics are sometimes called ‘principles, as they're not meant to dictate behaviour, 
but to ‘define general and fundamental propositions that affirm the tents of the 
profession, which can be adopted to guide action in a wide variety of specific 
settings’ (Colwell-Chanthaphonh & Ferguson 2006: 116-117). Failure to meet 
an organization's code of ethics has few, if any, consequences under an aspira- 
tional system. It is assumed that violations will be handled more formally by 
universities and employers. This is sometimes sufficient, but not always. 

Within archaeology it has historically been the case that ethical behaviour is 
believed to be assured by an archaeologist's participation in professional mem- 
bership societies and that the field should set its own standards of acceptable 
and unacceptable practice. Aspirational codes of ethics typically fulfil this role, 
due to archaeology’s relationship with academic departments and organizations. 

In contrast to aspirational codes of ethics, ‘prescriptive’ codes of ethics 
are more like systems of rules. A prescriptive code of ethics specifically lists 
acceptable and unacceptable behaviours within the discipline. As an example, 
the Code of Conduct and Standards of Research Performance of the Register of 
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Professional Archaeologists (RPA) is purposefully prescriptive, listing what an 
archaeologist ‘shall’ and ‘shall not’ do (RPA 2018). These standards are enforced 
via a grievance process, which is overseen by an elected officer whose role is 
to handle, ‘allegations of violations of the Code of Conduct and Standards of 
Research Performance of the Register, in accordance with the Disciplinary Pro- 
cedures of the Register’ (RPA 2018). Prescriptive codes of ethics are more com- 
mon in organizations that deal with contract or commercial archaeology, such 
as the Register of Professional Archaeologists and the Chartered Institute for 
Archaeologists (CIfA). 

Because the ethics of digital archaeology are not provided as aspirational sug- 
gestions of ‘good behaviour’ and ‘bad behaviour’ nor as prescriptive directives 
of ‘shall and ‘shall not; the onus is on those who work in digital archaeology 
to more carefully consider how they view their ethical relationship with the 
three areas of digital archaeological practice: digital tools in archaeology, digital 
methodologies in archaeology, and the digital education of archaeologists. We 
will turn now to discussing each of these areas. 


Digital Tools in Archaeology 


One view of digital archaeology holds that the digital aspects of archaeological 
practice should be considered tools (Zubrow 2006). This view holds that the 
use of a program such as QGIS, for spatial and relational mapping, or a digital 
camera and Adobe Photoshop, for site photography and creating a digital site 
archive, or a laser scanner, to create point clouds for 3D modelling of historic 
structures, are tools, to be used by archaeologists to meet an end, but without 
any larger ethical implications. They are equated to the trowel, the measuring 
tape, and the Munsell Soil Color Charts, in that they are utilized for a specific 
function, and their digital nature is irrelevant to the impact of their use on the 
completed archaeology. 

While this view is not inherently incorrect, there is a danger in assuming that 
the ethics of using a digital tool are the same as the ethics of using a manual 
tool. Both digital and manual need to be considered in light of their ‘ethical 
burdens: An ethical burden is the weight an archaeologist must give to whether 
any given thing is ethically appropriate, or inappropriate. As an example, the 
ethical burden of a trowel is typically low for archaeologists; it is regularly a 
necessary tool to be used in the process of excavation. However, if the use of 
the trowel in an excavation would disenfranchise indigenous peoples through 
unwanted exhumations of human remains, the ethical burden of the trowel 
becomes much greater. 

A common problem with ethical burden and digital tools in archaeology is 
what is referred to as ‘black box’ technologies. A black box is an object, piece 
of software, or system in which the user can direct input but cannot examine 
or verify the processes that occur before the produced output. Some potentially 
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black-box digital tools in archaeology include the previously mentioned digi- 
tal photography, geographic information systems (GIS), and photogrammetry. 
Most of these tools are facilitated through proprietary software, where the code 
that creates the output cannot be viewed. For most archaeologists, even open 
source software packages, such as QGIS and R, are black boxes, as they are used 
without a full understanding of what underpins the packages. 

For those interested in further discussion of the concept of a black box in 
archaeology, the topic has been discussed extensively via discussions of Latour 
(1987), and more recently and applicably by Huggett (2017) and Caraher (2016: 
434). Caraher explains its use in digital archaeology as the result of: 


...growing pressures on both academic archaeologists and those in the 
field of cultural resource management to produce results at the pace of 
development and capital. In other words, as digital tools accelerate the 
pace of archaeological work, more aspects of archaeological practice 
become obscured by technology. 


Digital Methodologies in Archaeology 


Another view of digital archaeology holds that the digital aspects of archaeo- 
logical practice should be considered for their value as methodological and 
theoretical approaches (Perry & Taylor 2018). This view is concerned with 
how the digital is being deployed in research designs, and how digital archae- 
ology is meeting larger issues related to public outreach and interaction with 
marginalized and indigenous populations. Again, we turn to the concept of 
ethical burden. 

When considering digital archaeology as a methodological approach, the 
ethical burden occurs almost immediately, during planning at the outset of 
the project. Digital methodologies should be subject to the same level of ethical 
scrutiny as the use of any manual toolset or analog methodological approach. A 
series of simple questions, asked at the beginning of research design planning, 
may result in the addition, or elimination, of digital aspects of the project. 

For every tool under consideration, we should ask, ‘Is the use of this tool in 
a digital form adding value to the project that is balanced by the ethics of its 
use?’ For every methodological consideration, we should ask, ‘Is this approach, 
mediated digitally, fulfilling all of our needs for it, without adding undue ethi- 
cal burden or breach?' If the answer to either of those questions is no, the use of 
the digital form should be weighed against the analog form. 

Just because something can be accomplished faster, or easier, with a digital 
approach, doesn't mean that the ethics of that approach are equal! The ethical 
burden might be too high. Understanding how that burden is borne by meth- 
odological and practical choices is the responsibility of everyone on a project. 
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As an example, consider a project involving human remains. There are poten- 
tially widely different ethical considerations between an analog methodology 
and a digital methodology (Ulguim 2018). 

In an analog methodology, human remains are excavated, laid out (if possi- 
ble) for examination by an osteoarchaeologist, photographed, and either stored 
physically for future study or re-interred. The data that is collected from analy- 
sis of the remains is a tangible, physical dataset, and who can interact with both 
the remains and the dataset is access limited. The longevity of the dataset is 
determined by how well the physical medium in which it exists is maintained. 
In this situation, the ethical burden on those involved with the project is to 
fulfil their research agenda while treating the remains with dignity and to do so 
(if applicable) through consultation that respects the wishes and cultural rights 
of descendant populations. 

In a digital methodology, human remains are excavated, laid out (if possible) 
for examination by an osteoarchaeologist, photographed, scanned for 3D mod- 
elling, sampled for digitally mediated analysis, and either stored physically for 
future study or re-interred. Digital records are produced of the remains, and 
digital copies may be created of the remains to be manipulated for methodo- 
logical testing models and to be made available to the public via outreach and 
museological interactives. The data that is collected from the analysis of the 
remains is a collection of digital files, and though who can interact with 
the remains is access limited, the digital files are often distributed more 
openly. The longevity of the dataset is determined by how dispersed the digital 
files are and how long the digital formats in which they are stored are viable, 
technologically. In this situation, the ethical burden on those involved with the 
project is the same as in an analog methodology, but with the added burdens 
of negotiating the potential differences in views towards digital permanence by 
indigenous populations and marginalized populations. 

No widespread study on how indigenous groups view their rights regard- 
ing digital archaeology related to their ancestors has been undertaken within 
archaeology and the ethics of digital archaeology related to human remains is 
being determined largely by non-indigenous archaeologists. A special issue of 
Archaeologies (Alfonso-Durruty et al. 2018; Hassett et al. 2018; Hirst, White & 
Smith 2018; Ulguim 2018; White, Hirst & Smith 2018) is the most thorough 
discussion to date concerning these issues. 


Digital Education in Archaeology 


Though archaeologists differ in their views of digital archaeologies as tools or 
methodologies, they share a common foundation in the process of education 
that leads to their professionalization within the discipline (Shaeffer 2016). 
How digital archaeology is conveyed to students, whether as tool or methodo- 
logy, is arguably less important than that it is being conveyed to students, and 
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that it is preparing them for the reality of the digitally mediated archaeology 
that is modern practice. Problems occur, however, when students are taught to 
use digital tools without teaching the accompanying ethical consideration of 
those tools (Dennis 2020). Students need to be educated in a process of ethical 
questioning concerning their digital outputs and in the resources available to 
address those questions. 

The practical issues that professional archaeologists encounter, both in aca- 
demia and commercial practice, are common to issues encountered by student 
archaeological researchers. These include issues in approaching and consult- 
ing with the public, issues in utilizing technology and digital tools, issues in 
decision-making concerning data storage and deposition, and issues in pub- 
lishing and outreach, amongst others. All of the ethical issues present in these 
situations for more advanced practitioners are there for student researchers, 
but students must frequently negotiate these ethical concerns without recourse 
to networks of colleagues for consultation, or professional memberships for 
guiding principles. 

Graduate students are often asked to consider the ethics of their practice 
through submissions to ethics review boards; however, that is frequently the 
first time that students encounter practical ethics in archaeology. Increased 
attention to undergraduate and entry graduate student-level interactions with 
ethics in digital archaeology, combined with an increased focus on the ethics 
of digital archaeology among those tasked with teaching students, is necessary 
to create a corpus of practitioners who are fully versed in the ethics of their 
profession. 

As students, and as educators, we have a shared responsibility to push the 
discipline of archaeology forward into a more equitable, ethical practice. Part 
of this shared responsibility means publishing the whys and hows of the digital 
practice we undertake. There is little peer-reviewed literature around teaching 
ethics to archaeology students concerning digital applications, which means 
that educators have few sources to draw on to inform their teaching practice 
and to share with students to demonstrate best practices. Notable exceptions 
to this include Perry (2018) on humanizing digital archaeological and herit- 
age practice, Graham (2016) on the creation of digital humanities notebooks, 
the work of participants in MSUDAI (2015) on collaborative cohorts in digital 
archaeological projects, and Cook (2018) on working with students to create 
ethically grounded digital exhibitions for museums. 


Conclusions 


Though the considerations of ethics within archaeology have arguably become 
more robust through the disciplines evolution, and digital archaeology has 
become more standard practice within the discipline, digital tools and digital 
methodologies have yet to be synthesized fully into archaeological discussion. 
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Alongside this, the pace of adoption of digitally centred archaeological data 
and digitally facilitated archaeological practice has not been met by the adop- 
tion of discipline-wide standards related to archaeological ethics. The result of 
this mismatch in ethics and practice shows itself most clearly in the pedagogy 
of digital archaeology, where little literature exists to educate newly inducted 
archaeologists who utilize digital forms. 

Digital archaeologists, and indeed all of us who are engaged with archaeol- 
ogy as professionals, have a responsibility to consider the ethical burdens of our 
research, as well as the tools and methodologies that we utilize to accomplish our 
knowledge production goals. It will be only through a shared effort that digital 
archaeology will come to be on par in terms of ethical consideration with estab- 
lished practices in excavation, analysis, publication, outreach, and education. 
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Abstract 


Evaluation in virtual heritage is concerned with learning about and assessing 
the extent to which an interactive system offers a satisfactory user experience 
(UX) and meets user goals and expectations. Evaluation in virtual heritage is 
an empirical process of research, which reaches for conclusions about the qual- 
ity of a system by observing, measuring (aspects of), and interpreting the UX. 
It is inherently a complex activity that requires careful planning and selection 
of methods. It does not rely on underlying technology; however, adaptations of 
process and methods must be made to allow for results and feedback in 
context. Therefore, it must be designed so that it is useful, reliable, valid, and 
productive. Evaluation methods and processes are of interest to both cultural 
heritage (CH) professionals and technology designers, who aim to provide sys- 
tems that address the widest range of potential users. This chapter discusses 
basic concepts, processes, and empirical evaluation methods in virtual heritage, 
with examples. 
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Introduction 
Virtual Heritage Technology 


An increasing number of interactive systems aim to enhance the user expe- 
rience (UX) of visitors at CH places and sites including museums, exhibi- 
tions, archaeological places, historic cities, or settlements. These systems have 
various goals, such as information presentation, learning, visitor engagement 
(when users assume roles and pursue learning goals), digital representation, 
preservation, or reconstruction of monuments, sites, or the like, combined 
with gamification elements or developed into a game. They are developed 
with contemporary and emerging technologies such as interactive walls, tables 
and surfaces, virtual/augmented/mixed reality (VR/AR/MR) systems, 3D vir- 
tual worlds, mobile location-based services and games, and so on. In a recent 
review article, Nikolakopoulou and Koutsabasis (2020) reported on the most 
common interactive technologies and interaction styles of virtual heritage: 3D 
game engines, mobile technologies, kinaesthetic interaction, physical comput- 
ing, VR, and AR. 


Importance of Evaluation of Virtual Heritage 


From a design and a user-centred perspective, evaluation in virtual heritage is 
concerned with learning about and assessing the extent to which an interactive 
system offers a satisfactory UX meeting user goals and expectations, adapting 
ideas and methods from fields such as human-computer interaction (HCI) and 
interaction design. 

The evaluation of virtual heritage leads to findings and recommendations 
about user acceptance, which interest both CH professionals and technology 
designers. Evaluation in virtual heritage is an empirical process of research: 
it reaches conclusions about the quality of a system by observing, measuring, 
and interpreting (aspects of) the UX. An indicative list of empirical evaluation 
methods includes observation, interview, user testing, field testing, field stud- 
ies, questionnaires, surveys, and diary studies. Evaluation concepts, methods, 
processes, and tools are generic: they are independent of interactive technol- 
ogy; however, some adaptations or specifications may have to be made. Fur- 
thermore, to achieve useful recommendations for (re)design, the affordances 
of interactive technologies must be considered. 

The purpose of this chapter is to highlight important concepts and pre- 
sent practical guidelines for the evaluation of virtual heritage. Here, the term 
virtual heritage denotes any type of computer-based interactive systems that 
promote CH. A visual model of the main concepts discussed is depicted 
in Figure 21. 
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Evaluation Process: Empirical and Iterative 


Evaluation in virtual heritage is an empirical process of research; it reaches con- 
clusions about the quality of a system by observing, measuring (aspects of), 
and interpreting the UX, which consists of various elements that arise during 
interactions such as usability, accessibility, engagement, sensitization, findabil- 
ity, learning effect, and so on. Empirical evidence, that is, the record of ones 
experiences, can then be analyzed quantitatively or qualitatively. Users are the 
most important factor in an evaluation process: they do not have to be many, 
but good representatives of real users representing a wide range of the intended 
audience or experts (in the case of system inspections). 

Evaluation in virtual heritage essentially happens iteratively during system 
development, to feed through the design, and operation, to help reflect on 
experience and impact. At design time, evaluation is exploratory, such as seek- 
ing required features. At the prototyping stage it is formative, generating issues 
for redesign. At the piloting or operation stage it can be summative, reaching to 
conclusions about the outcome. Each iteration differs in terms of the technical 
maturity of the system, the process and intended outcomes, the intended par- 
ticipants’ (evaluators’) profiles, and data collected and processed. 


Dimensions of Evaluation 


The evaluation of virtual heritage usually emphasises one or more dimensions 
that arise during interaction with technology. In the review of 83 evaluation 
studies, Nikolakopoulou and Koutsabasis (2020) identify the main dimensions 
of evaluations in: 


e User experience (19.9% of studies). 
e Usability (19.1%). 

e Perceived usability (8.5%). 

e Engagement (7.1%). 

» Learning (6.4%). 


In this section, we briefly present some of these dimensions, for which there 
is a large corpus of background work in many fields related to humanities and 
human sciences, culture, design, and technology. These dimensions are often 
not seen in isolation in evaluation of virtual heritage but in combination with 
each other and others. 


User Experience 


UX is a general, fluid, changing, personal, and subjective concept, with many 
definitions. According to ISO 9241-210 (Ergonomics of Human-System 
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Interaction), UX is ‘a person's perceptions and responses that result from the 
use or anticipated use of a product, system or service: According to Norman 
and Nielsen Group, UX ‘encompasses all aspects of the end-user’s interaction 
with the company, its services, and its products. Many models of UX have been 
proposed, such as those of Forlizzi and Battarbee (2004) and Karapanos et al. 
(2009). It is now widely agreed that UX incorporates pragmatic and hedonic 
product qualities: the former refers to the utility and usability aspects, while the 
latter consider the aspects of pleasure and emphasize stimulation, fun, identifi- 
cation generated by the use of a product or system. 


Usability 


According to ISO 9241, usability is 'the degree to which a product can be used 
by specified users, to achieve specified goals, with effectiveness, efficiency, and 
personal satisfaction, in a specified context of use’ Usability professionals have 
developed a whole corpus of evaluation methods and techniques (e.g., Nielsen 
1994; Cairns & Cox 2008), and they all share the following (Lewis 2014): 


a) a careful plan of study, including initial instructions and debriefing 
protocols; 

b) participants who are members of the population of interest; and 

c) appropriate tasks and environments. 


By definition, usability evaluation refers to performance and preference; the 
former is measured by metrics such as task success, task time, and errors, while 
the latter is obtained directly or indirectly via interviews, observation, ques- 
tionnaires, and so on. UX evaluation builds on usability methods or includes 
them as an essential part of the process (Albert & Tullis 2013). 


Engagement and Flow 


There are many definitions of engagement in HCI. According to O'Brien and 
Toms (2008), ‘engagement is a quality of user experiences with technology that 
is characterized by challenge, aesthetic and sensory appeal, feedback, novelty, 
interactivity, perceived control and time, awareness, motivation, interest, and 
affect. According to Doherty and Doherty (2018), engagement is often related 
to flow theory, which posits the existence of a state of optimal and enjoyable 
experience characterized by challenge, immersion, control, freedom, clarity, 
immediate feedback, temporal insensitivity, and changes in one's sense of iden- 
tity (Cowley et al. 2008). In virtual heritage evaluation, flow has been explored 
in CH projects of mobile museum narratives (Roussou & Katifori 2018). 
Engagement has been measured (Doherty & Doherty 2018) with various tech- 
niques and tools, including questionnaire, behavior logging, observation, task 
outcomes, interviews, eye tracking, and the like. 
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Presence 


The concept of presence originates in telepresence research. Minsky (1980) 
broadly described it as the feeling of being present (by appearance) or having 
an effect at a place other than their true location via technology mediation. 
Since then the concept of presence has been extensively discussed in computer- 
mediated communication (CMC) and VR research, where it has been defined 
(Lombard & Ditton 1997) as ‘the feeling of being there’ or ‘an illusion that a 
mediated experience is not mediated’ In virtual heritage, the concept of cul- 
tural presence has been proposed (Pujol & Champion 2012), as the feeling of 
‘being - not only physically but also socially, culturally - there and ther. 


Learning 


There are many definitions of learning in general as well as in relation to tech- 
nology as a mediator or to CH. Relevant to virtual heritage is non-formal and 
informal learning that is pervasive, self-regulatory, active, and participatory,’ 
in contrast to formal learning processes involving tutors, learning goals, and a 
classroom setting. Technology-mediated learning is typically assessed in com- 
parative evaluations in a control versus test group of learners. 


Accessibility 


According to ISO 9241, accessibility refers to the usability of a product, system, 
service, or facility to people with the greatest extent of abilities. Thus, it can be 
evaluated with the same methods as those of usability, provided that people 
with disabilities are included. In addition, the accessibility of mainstream tech- 
nologies has been specified into detailed guidelines that translate to technical 
features to which an accessible system must comply with. This is particularly 
relevant for the Web, where there are many open standards which promote 
Web Accessibility. 


Evaluation Approaches 


Main evaluation approaches can be broadly classified into inspections, labora- 
tory tests, and field studies. 


Inspections 


An inspection of virtual heritage typically takes place during the design pro- 
cess, by experts who experience the system, when this is not yet fully functional 
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or operable. Inspections are productive processes, but unless a systematic pro- 
cedure is followed, it can easily get out of hand. 

Inspection methods have been proposed in HCI several years ago; two of the 
most common are the cognitive walkthrough (Mahadoty 2010) and heuristic 
evaluation (Nielsen 1994). The latter includes: 


e A set of guidelines or heuristics to which the system must comply with, 
such as system visibility, consistency, persistence, and recovery from errors. 
e A systematic common process followed by evaluators, which: 
e firstly, includes atomic inspection and identification of findings matched 
to the heuristics list, and 
» secondly, requires from evaluators to meet and combine their findings 
into a common list with severity marks and priorities. 


In virtual heritage, we neither have well-established sets of guidelines or heu- 
ristics nor can we easily identify a small number of experts that possess all 
required knowledge and skills to perform an inspection; thus, inspections are 
uncommon in scientific literature. Until we have these resources, inspections in 
virtual heritage are typically performed in practice by expert evaluators about 
aspects of the UX (e.g., usability, accessibility, presence), content (e.g., museum 
curators and other CH professionals), and interaction technology. To put some 
method in the process, these experts should work in a systematic manner, per- 
haps with a set of goals and guidelines that are produced in-context, to produce 
a common set of design recommendations that the system under evaluation 
must conform to. 


Laboratory Tests 


Laboratory testing takes place at the final stages of the design, or a design itera- 
tion. It involves test users who provide direct and indirect findings and feed- 
back. The process has been detailed in many HCI textbooks (e.g., Albert & 
Tullis 2013; Cairns & Cox 2008; Nielsen 1994). It is generic and can be adapted 
considerably depending on target technology, availability of users, time, and 
other resources; it also requires considerable preparation. 

It essentially involves the following steps: 


(a) defining the place, time and prototype for evaluation, 

(b) recruiting representative users, 

(c) defining user tasks for testing, 

(d) defining measures and data analysis, 

(e) conducting the test — for each user the process is identical and data 
collection takes place, 

(f) summarizing the test, 

(g) data analysis and reporting. 
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It is the most common method for general evaluation of interactive systems, 
where the evaluation may be repeated in an iterative development process. 
In literature about virtual heritage evaluation, it seems that most approaches 
are laboratory tests of various configurations (Nikolakopoulou & Koutsabasis 
2020). Testing is often comparative, between/among alternate systems, such 
as reported by Jylha et al. (2015). They evaluated their wearable interface for 
exploring urban POIs by assigning 12 users in two groups: a baseline group 
that used a mobile phone app and the test group that used the wearable app. 
Comparative testing can also be performed between/among user groups, for 
example, expert/novice users (e.g., Panayiotou and Lanitis 2016, on 3D ani- 
mated paintings) or between adults/children (e.g., use of a gesture-based app of 
Koutsabasis and Vosinakis 2016) or within system configurations. It may also 
be formative, that is, emphasizing on qualitative analysis and generating design 
recommendations, or summative, that is, emphasizing quantitative analysis 
and statistical testing. 

It might also happen in online platforms, if the system under evaluation per- 
mits, although the researcher loses contact with users; or it might occur in the 
field, which is useful but must be well planned so that the process is controlled 
for all users. 


Field Studies 


A field study is a general process of observing actual users interact with 
technology located in the real place, time, and context, gathering data and 
reaching conclusions about aspects of user interactions with technology. 
Field studies are invaluable for assessing the quality and impart of virtual 
heritage. They may be employed not only to assess dimensions of user inter- 
action with a recently introduced system as well as to evaluate existing 
interactive technologies in cultural places and sites. Essentially, a field study 
boils down to sophisticated observation and technology-mediated record- 
ing of user activity. 

There are several field studies in virtual heritage literature, such as, for exam- 
ple, the work of Rubino et al. (2015); they integrated a location-based mobile 
game in the museum visit and evaluated visitors’ behaviour and learning by 
inviting them to play the game during their visit. Or consider the work of 
Caggianese et al. (2018), who installed a gesture-based interactive holographic 
projection in the museum and evaluated visitors’ engagement and the attrac- 
tiveness of the system. While some aspects of the study can be controlled, espe- 
cially the tasks that users or visitors are asked to perform, it is still a form of field 
testing. Another example of a field test is the work of Koutsabasis and Vosina- 
kis (2018), who invited teenage museum visitors to digitally sculpt Cycladic 
figurines in an interactive kinaesthetic game. 
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Evaluation Methods and Techniques 


Various methods and techniques for the evaluation of virtual heritage have been 
employed. In the review by Konstantakis and Karidakis (2020), they examine a 
long list of evaluation methods. According to the review of 83 cases of virtual 
heritage evaluations (Nikolakopoulou & Koutsabasis 2020), most make use of 
questionnaires (39.996), observation (1996), and interviews (16.196). 


Questionnaires 


Questionnaires may be either standardized or developed by evaluators. The 
former has gone through the process of psychometric validation for several 
dimensions of evaluation, like UX (e.g., User experience Questionnaire, Strepp 
2017), usability (e.g., System Usability Scale, Brooke 2013), presence (Witmer 
& Singer 1998), and so on. 


Observation 


Observation can be organized into many metaphors (Shafer 2009), such as: ‘fly 
on the wall’ (observe unnoticed), ‘shadowing’ (discretely follow a user), and 
‘secret agent’ (play the role of a user). 


Interviews 


They can take many forms, such as structured, semi-structured, directed story- 
telling, group interviews, site walkthroughs, contextual inquiries (Bayer & 
Holtzblatt, 1995), and so on, and can happen in the field (preferably), in the 
office, online, or by phone. 


Intervention Protocols 


These protocols are mainly employed in user testing, besides post-hoc inter- 
views, and they usually include (Van Den Haak 2003): 

a) concurrent or retrospective think-aloud, that is, when users are 
encouraged to speak their thoughts about their interactive experi- 
ences, as in Correia et al. (2014), who encouraged users to think aloud 
during the use of their interactive installation, which enabled them to 
reconstruct medieval illuminations of old books, and 

b) constructive interaction, that is, testing in pairs of users who inter- 
act with each other, which is a very productive method for formative 
assessments (Koutsabasis et al. 2007). 
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These are all empirical methods of research and therefore 
a) they all yield knowledge and results, with a limited scope, 
b) they can be all useful, but their application can yield errors, 
c) errors of one method can be corrected by another, and 
d) different methods must be combined in a comprehensive evaluation. 


Other General Issues 


Important issues to consider in the evaluation of virtual heritage include: 


User representativeness 


The selection of representative users is the most important aspect for a success- 
ful and valid evaluation. Preferences, knowledge, and skills vary among people; 
therefore, we often observe significant variability in performance and prefer- 
ence. Thus, it is important to recruit representative users, if it is not possible to 
recruit people who will actually use the system. 


Ethical issues and privacy 


Any empirical evaluation requires the participation of users, who must always 
participate freely and willingly. Cairns and Cox (2008) use the acronym VIP 
(vulnerable participants, informed consent, privacy) to denote the three major 
ethical issues that must be addressed at any evaluation process. Another ethi- 
cal issue, from a scientific perspective, is about evaluation data, which must be 
either open or readily available to potential reviewers or colleagues. 


Evaluator knowledge and skills 


Currently, evaluation of virtual heritage is performed by researchers from fields 
including HCI, cultural heritage, design, psychology, learning, and the like. We 
do not foresee that this will change in the near future because evaluation is a 
holistic process that requires multifarious expertise taking into account tech- 
nology affordances and digital curation, and can provide insights about digital 
content, user interface, and interaction design, software engineering, usability, 
and user experience. 


Planning the evaluation and managing trade-offs 


An evaluation of virtual heritage requires planning in terms of goals, approach, 
participants, measures, place, and time. As with any practice-oriented activity, 


Evaluation in Virtual Heritage 125 


planning must consider practical constraints and trade-offs regarding availabil- 
ity of main resources such as time, users, and technology. 


Conclusion 


Virtual heritage is a highly suitable domain for contemporary interactive 
technology development. Cultural heritage organizations are addressing the 
widest possible range of potential visitors, with an emphasis on younger people 
and children, who are attracted by interactive technology. Visitors of cultural 
heritage sites are interested in maximizing their experience in terms of sensi- 
tization and learning, mediated by technology. Technology developers pursue 
novel designs in virtual heritage, which provides a challenging context welcom- 
ing novel interaction with technology for a wide range of user requirements. 
Evaluation of virtual heritage can be performed with various approaches and 
methods to reflect several dimensions of UX. If it is conducted with care and 
rigour, evaluation can ensure that the requirements of interested parties are 
incorporated into the technology solution. 


Notes 


! Free icons obtained from https://www.flaticon.com/ 

? https://museum-id.com/informal-learning-museums-opportunities-risks 
-gina-koutsika/ 

? https://www.w3.org/WAI/ 
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Abstract 


This chapter examines why we wish to preserve heritage objects and practices 
via virtual heritage, and why the issue of authenticity is so important here but 
so seldom addressed. If we could give criteria to select and to create useful and 
even authentic-oriented virtual heritage projects, what would they be? Or are 
there methods and solutions out there waiting to be discovered? 


Why Aren't 3D Models Used, Wanted, or Archived? 


If virtual heritage is a preservation medium (and not just a communication 
medium), an immediate question might well be: why are virtual heritage pro- 
jects worth saving? Are they worth saving? In the case of 3D and game environ- 
ments, some have suggested even knowing what they are to be used for can be 
extremely unclear. 
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Digital Heritage - The Vanishing Virtual 


Are they being saved? Digital Humanities is/are all around us, yet 3D models, 
especially 3D cultural heritage models, are seldom discussed as part of schol- 
arly arguments (Di Benedetto et al. 2014; Huggett 2012), nor are they valued as 
research output, made accessible to the wider public (Talboom & Underdown 
2019), or generally evaluated for significance, engagement, or impact. 

UNESCOS Charter on the Preservation of the Digital Heritage warned of the 
increasing risks to heritage (UNESCO 2009): 


... the rapid obsolescence of the hardware and software which brings 
it to life, uncertainties about resources, responsibility and methods for 
maintenance and preservation, and the lack of supportive legislation. 


According to Hal Thwaites (2013), *...digital heritage [is] disappearing faster 
than the real heritage’ and there is urgent need for greater accessibility, consistent 
interface design, global infrastructure, archival standards, and ongoing curation. 


Formats and Platforms 


Although there have been useful recent surveys on 3D model formats (Fernie 
2020) and an upcoming survey on digitalisation quality (Iliofotou 2021), we 
still lack sufficient surveys on required features and procedures for 3D file for- 
mats, their relative dependability, accessibility, playability, graceful degrade- 
ability, scalability, and range of device-suitability, even if their long-term cost 
and proprietary status can be determined. The problem is not through a lack 
of 3D formats, there are over 140 3D formats (McHenry & Bajcsy 2008), but 
they don't always allow for ease of use across various applications (Neamtu et 
al. 2014; Tsiafaki & Michailidou 2015). 

Another immediate problem is access. In previous work, with Dr Hafizur 
Rahaman (Champion & Rahaman 2020), we reviewed virtual heritage platform 
hosting solutions. None offered all of the basic interactive features we thought 
would be most useful for virtual heritage. The commercial Sketchfab website 
offered the most features and hosted the most models, with a variety of import 
and export formats. Common platform formats were FBX (Autodesk), glTF/ 
GLB, DAE (Collada), and OBJ, but there was also USDZ (a 3D file format that 
displays 3D and AR content on iOS devices), and application-native formats 
(such as UNITY). U3D and VRM/X3D did not seem to be popular formats on 
these platforms. IIF 3D is an interesting 3D format but is still under develop- 
ment and discussion. 

We have also surveyed the availability of 3D models and related media 
(Champion & Rahaman 2019) in virtual heritage conference presentations. 
From a group of 1483 conference papers over the period of 2012 to 2017, we 
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selected 264 of the total papers published in VSMM, CAA, CIPA, EuroMed, 
and Digital Heritage Congress. Only 17.9% referred to and contained images of 
3D assets or 3D digital models. Only nine papers contained accessible 3D assets 
or 3D models. Nineteen articles contained external web links to 3D models but 
not a single link worked on the final day of our survey: 1 September 2018. 

UNESCO (UNESCO 2016) provides several criteria to ensure the preserva- 
tion of documentary heritage (including multimedia), reflected in its Memory 
of the World program (https://en.unesco.org/programme/mow/). A document 
itself ‘is preservable and usually moveable ... [and] ... content may comprise 
signs or codes (such as text), images (still or moving) and sounds, which can 
be copied or migrated? UNESCO states documentary heritage ‘should be per- 
manently accessible and re-usable by all without hindrance ... with due respect 
to and recognition of cultural mores and practicalities’ and suggests that every 
country (state) <.. should develop training and capacity-building schemes as 
appropriate to ensure the identification, preservation and access to documen- 
tary heritage training: 


Use and Reuse 


To fulfil these objectives, UNESCO’s Guarding Against Loss Of Heritage 
states we must build cooperation between all relevant public and private sec- 
tors, develop training and research, and encourage research organizations to 
preserve research data. One extra ingredient is needed; building on the FAIR 
Principles (https://www.go-fair.org/fair-principles/), ideally the project pro- 
vides not just Findability, Accessibility, and Interoperability, but also the Reuse 
of digital assets. 

Even academic reuse is uncommon. Although there have been virtual herit- 
age conferences such as Virtual Systems and Multimedia, academic proceed- 
ings do not host the 3D models. There is an increasing number of academic 
journals providing an online display of 3D models, but they are not archives. 
They dont easily allow for maintenance and upgrading of the components. 

Is the whole issue of 3D model preservation just too difficult? Is it really out- 
side the scope or capacity of major players in the GLAM (galleries, libraries, 
archives, and museum) sector, or are these 3D models just not engaging to the 
public? Although the London Charter (Denard 2016) and various UNESCO 
Charters (UNESCO 2009, 2015) advocate the use of paradata and measures to 
counter technical obsolescence and physical decay, one major element is miss- 
ing. Without doubt, we need to preserve and integrate 3D/multimedia; pro- 
vide access and record the ownership of models, sites, and paradata; develop 
suitable guidelines and shared procedures; collate and distribute standard- 
ized evaluation data; and incorporate data tracking audience engagement 
and feedback. But we are still sidestepping a more fundamental challenge: the 
user experience. 
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We also need to develop incentives - incentives for developers to provide 
showcases; ongoing funding through critical mass; use, reuse, and critical feed- 
back in teaching; delivering to a wider audience; providing prizes, awards, or 
other recognition for technical collaborators; long-term depository citation 
and dynamic linking; modification of creative commons specifically for 3D 
heritage models, components, and sites; and providing a variety of level of data, 
access, or components. 

There are research consortiums that handle their own online and archival 
functions, such as Europeana (Ubik & Kubista 2017) and the Smithsonian 
Institute (with their X3D beta), but academic research collaborations such 
as ARIADNE, 3D Icons, CARARE, and EU Epoch relied on regular research 
grants to continue or transferred their tools and case studies to their part- 
ners. They can provide expert and scholarly information on these models and 
sites. However, they are not geared towards use and reuse, streamlined 
and shareable across a variety of platforms with a standardized interface. 
Sketchfab (https://sketchfab.com) is one, and the Smithsonian (Daher 2020) 
and CYARK now use it. 

Commercial online hosting solutions like Sketchfab are designed for pres- 
entation rather than for preservation and dont require scientific overview 
(Statham 2019). Conversely, for traditional libraries and museums, digital 
archives are relatively new and not particularly suited for 3D models. The 
GLAM sector lacks the requisite financial resources let alone enough time or 
adequate staffing. Unfortunately, virtual museums do not curate and preserve 
the models. And, according to Birrell (2019), many projects lack clear plans or 
even an understanding of who the end-users are (Bettivia 2016). 

Communal groups, such as Micropasts (https://crowdsourced.micropasts 
.org/), by contrast, rely on volunteers. What they have in common is criti- 
cal mass, features, and shareability. Micropasts has an added attraction, 
you can collaborate on archaeological problems and upload your own 
additional media. 

Given virtual heritage implies the use and reuse of projects, I suggest there 
are at least six components required for preservation: 


1. The dataset (2D, 3D, textures, sounds, scripts, etc.) of the virtual herit- 
age itself. 

2. The paradata that helped the research and development of the virtual 
heritage project. 

3. The authorship, institutional links and accreditations, and teamwork. 

4. The intentions of the authors. 

5. The metadata and system structure and any relevant classification 
data. 

6. Evaluation data (audience tracking, usability studies, audience 
engagement results, and an attempt to capture usable and useful audi- 
ence experience and feedback). 


Preserving Authenticity in Virtual Heritage — 133 


Are Virtual Heritage Models Authentic? 


Must we slavishly copy, preserve objects in glass or cotton wool, and reproduce 
them with the most expensive equipment with the greatest possible digital 
file sizes? Every extra byte of a digital file requires hosting somewhere. Every 
computer server accessing the internet is powered, with a related environ- 
mental cost (Griffiths 2020). Despite improvements in the features, capacity, 
reliability, and precision of 3D formats, there is another, deeper underlying 
issue: what are these models preserving and what are they communicating to 
their audience? 

The question of authenticity is not merely a question regarding the model, 
but also a question on the intentions of the modelmaker. Arguably the authen- 
tic is not only the saved object and the intentions of the preserver or model- 
maker, but also the relationship people had with the object and the care shown. 
Care cannot be maintained behind glass even if artefacts left behind are as 
close as we can get to the past; they are inscrutable and mysterious ciphers to 
that culture. And the authentic is not merely the exterior, the similarity to the 
past, because we typically don't know the past. But the work and respect that 
went developing, maintaining, and handing down that cultural heritage object, 
belief, or performance is authentic. Can virtual heritage provide some sense 
of the relationship past people had to distant, remote, half-remembered, and 
disappearing places? 

The user experience relies not only on technology and access to that tech- 
nology but an experience of cultural heritage that is considered in some sense 
authentic and meaningful by the participant (Gilmore & Pine 2007; Van 
Balen 2008). 

In Authenticity: Depicting the Past in Historical Videogames, James Sweeting 
noted that authenticity could refer to “of undisputed origin of authorship’, or in 
a weaker sense a “reliable, accurate representation” (Sweeting 2019). Authen- 
ticity could, however, mean more than authorship or accuracy. Could the vir- 
tual heritage experience also be viewed in terms of authenticity? 

These considerations lead us to the following interpretations of ‘authentic’: 


1. ‘Authentic’ might mean the results are seen by shareholders or experts 
as belonging to them or are accurate renditions of the original source 
materials. 

2. ‘Authentic’ could be the interpretation by the original shareholders 
that the depictions or recreations are as per their trans-generational 
practices and beliefs. 

3. Parallel to option 2, ‘authentic’ can refer to the authenticity of what is 
made, and how is it made (is the creation process true to tradition? 
Does it show care?). 

4. Or ‘authentic’ could be an allusive but illusive relation to the experi- 
ence, or expectation of character, by a shareholder or domain expert. 
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We can focus these different strands of authenticity in terms of virtual heritage. 


1. How an object is made. 

2. Who owns or connects to the heritage depicted. 

3. Who understands the culture from which an object is made or intan- 
gible heritage is performed or shared, or the experience itself. 


These are all different but intertwined notions of authenticity. So, when virtual 
heritage is seen as authentic (if indeed, it is ever viewed through the lens of 
heritage), it could be seen as the safeguarding of ownership, identity, memory, 
craft, and art practices and so on. ‘Authentic could indicate ownership, of cul- 
tural affinity, but also authenticity in terms of where it is made or how it paral- 
lels experience. Authenticity leads to more questions, but when detected it can 
lead to increased engagement (Bunce 2016): 


. .. visitors who perceived the rabbits as authentic were more likely to 
ask a question than those who judged them as inauthentic. Perceived 
authenticity also promoted more why questions. 


Given the above and considering there are cultural distinctions in the under- 
standing of ‘authenticity, I suggest the goals of virtual heritage preservation 
are to preserve: 


1. Projects and related data: the virtual heritage projects themselves. 

2. Ideals: through raising awareness of the original materials and intan- 
gible assets or practices through the depiction and safeguarding of 
virtual reality and related digital media. 

3. Specific generative and transmissive knowledge: The cultural knowl- 
edge, and current understanding ofthe specific cultural knowledge that 
gave the development and maintenance of cultural heritage meaning. 
For example, virtual heritage is not a collection of mere objects. It is 
the learning mechanism that transmits values of cultural heritage. 
Even with physical heritage sites, we preserve the material heritage to 
safeguard the related intangible heritage. 


Conclusion 


The Nara Charter (ICOMOS 1994) stipulates that authenticity is an essen- 
tial component of cultural heritage. If we follow UNESCOS stipulation of the 
importance of cultural significance, what is worthy of preserving is the useful, 
the unique, the memorable, and the inspiring. So it seems we must aim to pre- 
serve ‘the authentic, and this implies an objective yet universal truth. Interpre- 
tations differ, and resources are limited. A related thorny question for virtual 
heritage might be in determining the purpose of a virtual heritage model in 
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terms of people's decisions, resources, and lives. The concept of authenticity has 
more recently been stretched and teased by theorists arguing over what herit- 
age can be saved and lived with (Harrison 2013; Holtorf 2018). 

How can digital heritage fulfil the noble aims of cultural heritage and the 
needs of society if it cannot even maintain, preserve, and sustain itself? There 
are too many divergent formats, no universal standards, missing case studies, 
and little agreement on protocols, standards, or parameters. Plus, the GLAM 
sector has increasingly limited resources (Münster 2019) to keep pace with 
changes in technology, hardware, applications, or social media trends. These 
problems are solvable but also depend on an improved understanding of 
authenticity, not only in terms of accuracy and precision and authorship but 
also in terms of illusive, perceived authenticity. 
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ENT 


Virtual heritage has been explained as virtual reality applied to cultural heritage, 

but this definition only scratches the surface of the fascinating applications, tools 

and challenges of this fast-changing interdisciplinary field. This book provides an 

accessible but concise edited coverage of the main topics, tools and issues in 
virtual heritage. 


Leading international scholars have provided chapters to explain current issues in 

accuracy and precision; challenges in adopting advanced animation techniques; 

shows how archaeological learning can be developed in Minecraft; they propose 

mixed reality is conceptual rather than just technical; they explore how useful 

Linked Open Data can be for art history; explain how accessible photogrammetry 

can be but also ethical and practical issues for applying at scale; provide insight 
into how to provide interaction in museums involving the wider public; and 
describe issues in evaluating virtual heritage projects not often addressed 

even in scholarly papers. 


The book will be of particular interest to students and scholars in museum studies, 
digital archaeology, heritage studies, architectural history and modelling, virtual 
environments. 
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